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ABSTRACT 

This survey of recent developments in educational 
program evaluation is intended for persons who commission, implement, 
direct, or carry out evaluation studies. The attitudes of government 
officials, educators, and researchers toward assessment and their own 
evaluation needs are discussed. Various approaches to evaluation are 
briefly described; the author emphasizes informal methods as opposed 
to standard psychometric measures. Instructions for estimating costs 
of an evaluation project are not provided, but suggestions for 
effective use of funds are included. Advice is given for evaluators 
in planning an evaluation study and negotiating an agreement with the 
monitoring officials. Hypothetical conversations between an official 
and a prospective evaluator are included as examples that will help 
the reader start an evaluation properly. (GDC) 
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PREFACE 



CERI, since its inception in 1969, has concerned itself with 
change in education; with the mechanisms that might bring it about, 
with the organisational structu-es that will nurture it, and with 
the nature of changes in content and method in the curriculum itself. 
Time and again one insistent question has emerged: how successful 
have the changes been? 

Ihe objectives of education ar^ so manifold, the means by which 
they are achieved so interrelated, and the manifestations of success 
so difficult to measure - even to observe - that evaluation has come 
to be seen as necessarily complex and often imprecise in its findings. 
Yet if sound Judgements are to be made, a sound basis of information 
is essential. 

This report clarifies and may throw light on the problems in- 
volved, and points the way to possible solutions. 

J.R. GASS 
Director 
Centre for Educecional 
Research and Innovation 



BACKGROUKD 



In November 1973 CERI launched a project on "New Approaches to' 
Evaluation of Educational Programmes". in the previous years most 
of the work on innovation in education ha ' posed the prbblem "how 
does the administrator know whether rr not an innovation, introduced 
experimentally, is worth generalizing?" It is this question that 
the project has made central to r.ts work. The stress has been less 
on the major policy changes, such as the structural alteration of a 
system to comprehensive education, and more on the consequent inno- 
vations in content, method and organisation in the educational pro- 
granimes that arise from them. 

Particular stress has been put on the process by which the 
administrator decides what information he wants and then commissions 
the necessary evaluation study. However, this has involved also a 
study of the differeny^styles of evaluation and some consideration 
of the methods by whi€h the results could be communicated. 

This report is the work of Professor Robert Stake of the 
University of Illinois at Urbana-Champaign aided by experts from 
many countries and by post-graduate students. He was able to spend 
a sabbatical year, preceding the writing of this report, in Sweden 
and the United Kingdom. To --elp in the process CERI organised an 
international Conference at Li^ge, hosted by the Belgian authorities, 
which used much of the first draft as background materials. A docu- 
ment, CERI/NAE/75.01 , which summarizes the work that went into and 
emerged from this Conference is available on demana. 

To be appreciated fully, this report should be read in con- 
junction with the document, free on demand from CERI, entitled 
•^Case Studies in the Evaluation of Educational Programmes". These 
were collected and edited by Professor Stake. 
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Chapter I 

THE PURPOSE AND LIMITATIONS OF THE REPORT 



This report has been prepared as a survey of the develcpments 
in educationpi progra.ame (1) evaluation in x.he middle 1970^*3. It 
is expected primarily to be of use to persons who will have responsi- 
bility for commissioning and implementing evaluation studies. Most 
of those people will be govi>rnment officials but some will hold posi- 
tions in schools, in industry, with philanthropic foundations or 
will be members of special councils. It will be their responsibility 
during the remainder of the 1970's to be the commissioners of evalua- 
tion, to provide the initiative and direction for evaluation studies. 

The report is written also for evaluators who will carry out 
the new studies. Many of them will not have previous experience in 
an evaluative study of national scope - there are not many people in 
the world who have. Some prospective evaluators will have had ex- 
tensive experience in educational research, and the differences be- 
tween research and evaluation may be difficult for them to understand. 
Some prospective evaluators will have had experience in teaching or 
£dministration, and their inclinations to move from evaluation into 
decision making for the programme may be a problem. They and other 
researchers and educators may find useful information in this report, 
but it has been written mostly with the education official in mind. 

- ' The i^reatest particular responsibility considered in this re- 
port is that of ?:etting the evalus*^.ive study started in a proper 
manner. The issues to be discussed by officials ana prospective 
evaluators prlr;r to making an agree^ient and the considerations that 
might be included soecifically in th^t agreement are given special 
attention , 

It is apparent that officials and evaluators have different 
expectations about what st'idiec can and should accomplish. Although 
thoir differences are never completely resolved, thorough discus- 
sions can ^lelp, It is not assumed that the contract that fixes the 
details most rigidly wiLi serve both parties best for substantial 



1) "Programme" in this report is used as a portmanteau word to cover 
the content, method and frame of a learning system. 



changes in expectation and need will oi"tp}. occur during the conaact 
of the st-udy. It ir t ho puroi^jie of this doc"'i< r z^, to alert the 
reader to seme ui' the ii^sur*? that may re."iu::*e planning and formal- 
agreement. 

There are», of course, many diiier-eni evaluation responsibilities. 
In this report attention har been '.-i r^ly ^::iven to the establishment 
of a large-scale, perhaps nnt ionril» t*-. nluative study, probably to be 
conducted once only rnthor ib^^r or r* rocurrinf basis. Mo el'fort was 
made here to exrimlne th? rrr>b.tfn;'. < f establishing': better procedures 
for monitoring ordinar^y c^; -.-..ions in the schools. No effort was 
made here to exaxir.e the pj-b'ems of prescribing evaluation require- 
ments tr. be carried out by i^-roject officers or their subcontractors, 
in special pro^^rammes r.ij;*n as in the Head Start Program in the U.S.A. 
or in the state-f'i\ ied innovritave projects in Hesse, West Germany. 

Emphasis is g'-ren to the evaluation of a particular programme, 
usually an already operating programme, rather than to the evalua- 
tion or a particular issue. Two examples of issues are changing 
selective and special schools into comprehensive schools for all 
pupils and uslntr a special alphabet to teach reading. • Issues are 
Important aspects of prof^ram^ie evaluation; but if the focus is on 
the issue rather than on the pro^^ramme, more traditional educational- 
research methods are likely to be appropriate. 

It is not su^";r•^ested in thi s report that education has not been 
evaluated heretofore. Ka. :i teacher, each administrator throughout 
the years has been evaluating. Informal, personal evaluation should 
be no less important at a time when formal evaluative research 
studies are becomln;^ Important. The new studies provide information 
to persons who are not in a position to make personal observations 
o^ who do not have (rood v;ays of combining; informal information from 
many schools. The new studies provide an additional kind of infor- 
mation about Inr.-'e programmes, potentially more accurate, more ob- 
jective, and more valid for the purpose on hand. The new studies 
are a response to an emerir^inpr need. 

Attendance and fisc^^l records, and in some countries pupil test 
records, have loru' been kept. These records are more transportable. 
They can to accumu La ted and exami ned by a d i stant educational of f i- 
ciai or t^rji^r-amrne spon^-jr. ;:'ut nuch records alone leave their 
I'eaders po irlv .^cf-.jf'^inted with the accomplishment and the prf)blems 
of the educati oaa 1 Tirorramme . 

rt \ Sw::i;,';;od thn .3II locrtl : n: 'jr:!)'^ t ; c.nCi r'±[ problems 

need the --ittenti'^n 0:' d:r.t.?nt officials. But some additional infor- 
mation i s sourh t . There i s a ou i ck 1 y inc rea s i nr; bel i ei' tha t more 
inform.ation on schori proceed i n/^;? anrl on the pvidence of stren/^tb 
and v;eakness of those proceed inrs lE esses! ial to the proper govern- 
ance of education. 



As this report has been written by an author whose experience 
is largely in one country for readers in many countries, and who, 
inevitat.y, has his own point of view, it has certain limitations. 

recognised that the need and responP.ibi I i ty will be dif- 
f-^^-.. each country, just as the history and present circumstance 
v:^', .jferer.^. in each. The American experience is unique - an 

-'.^.rv ^^-tnta". ive past and a most uncertain guide to the future. Yet 
it is. the experience cited mostly in this report - not chosen on its 
merit but because, like Mt. iCverest, it is there, and because it is 
the experience :nost familiar to the author and others who helped 
prepare this report. It is hop^O that the fullness of their de- 
scriptions will permit readers to ascertain the relevance of the 
findings, the models, and the n.Hvice contained herein to the present 
circumstances and to future evaluation work in their countries. 

In America formal educational programme evaluation has been 
^;rently influenced tay string professional commitment to standardized 
testing, by an expectation that local boards of education and parent 
j^roups will help determine instructional objectives and curricular 
priorities, and by the development of commercial agencies for the 
contracting of larf^e-sciUe studies such as the evaluation of federal- 
ly funded projects. Observers such as Ernest House and David Cohen {^) 
have recognised the highly political nature of the demand in the 
United States for evaluntlon of education. Such demands are to be 
heard wherever legislatures meet and are common phrasing in educa- 
tional tracts distributed both ty reform and establishmentarian 
groups. In recent years, particularly in the face of ballooning 
costs, demands for evidence of accountability of the schools to 
thei.- communities have bern widespread and even strident. Although 
many completed evaluation studies fell short of their promise, the 
demand for evaluation in the United States has not diminished. 

The author has worked with designs and procedures for evaluation 
at the local school-district level, at the state level, and at the 
lederal ]eveL for a little over ten yearr. His experience is largely 
limited to American work - and his orientation has been somewhat 
uncommon even there. As the listings of Section HI indicate, he 
has promoted a relatively naturalistic, process-oriented approach 
to evaluation. Readers who favour a more psychometric, experimental, 
or product-oriented approc.ch may find the report unduly biased. Tt 
is hoped that all readers will take these personal commitments and 
limitations into acoount and even be stimulated to further study of 



1) See Ernest R. House (Ed.), S chool Evaluation : The Politics and 
Pr^ocess, Berkeley: McCutchan, 1973; and David^ Cohen, 
..Politi cs and Research - Evaluation of Large-Scale Programs,' 
Review of Educational Research, 19^0, ^-O (2), 213-238. 
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their implications. A bibliography accompanying the matrix in 
Section III should be helpful in identifying the risks and potential 
payoff of this and alternative approaches to the evaluation of 
educational programmes. 

The limitations of this report cannot be listed in full, partly 
due to the finite length of the report; but one additional limita- 
tion requires mention. The report is limited by the point of view 
that responsibility for research upon educational affairs should be 
aggressively assumed by educators. Tais is not to say that citizens 
and elected officials should not investigate these affairs. This is 
not to say that scholars from the many learned disciplines should 
not study education. Nor should educational researchers disdain the 
use of concepts, methods, or advico of researchers from other fields. 
It is a point of view tnat assumes that educators must take primary 
responsibility for the *;valuation of educational programmes. 

The education profession ha i rather successfully maintained 
control of educational research. Few researchers are full-time 
teachers or administrators, but most consider themselves members of 
the profession rather than members of a discipline such as psychology 
or economics. Their work is generally in good repute - only a few 
critics find it essentially without merit. (1) It: is the educational 
researcher who is most frequently asked to direct an evaluation 
study, and it is he who writes most of the treatises on how it 
should be done. Those writings do not always inspire confidence 
that the profession can evaluate its own work. A v/idely respected 
American researcher commented :( 2 ) 

".-.we have reached the end of an era t.'iat should never have 
happened. There is nothing more tedious than having a collec- 
tion 01 miscellaneous essays, each one decorated with the 
author's "systematic" tabie or flowchart, ail taking in each 
other's washing. The point of diminishing returns in essays on 
educational evaluation was passed in 1968 or l970. 
"...none of us thought to ask whether educational evaluation 
was a field by itself distinct from other evaluation of social 
systems. There had in fact been a fairly substantial evalua- 
tion movement in the postwar years that was mostly in the hands 
of applied sociologists. Without nny insistence that they made 
great advances, I think it was s miijtake for us to proceed with 
a parochial discussion, to develop our own jargon » and to talk 
only with a small group of insiders. 



1) See Daniel Kallos» "On Educational Scientific f^esearch," Report 
No. 3b, Institute of Education, University of Lund, Sweden, 
April 1973. 

2) Personal correspondencf? t-.) the author, February 11, ^^7i\. 
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"...our isolation has been increasingly damaging as work in 
evaluation shifted in the middle '60s to the evaluation of pro- 
gram and policies, as di.otinct from the evaluation of types of 
classroom practice. ... Few of us have been able... to par- 
ticipate constructively in the debates about how future evalua- 
tion money is to be spent. And this is because that front has 
been manned by the sociologists, economists, and systems 
analj'sts. " 

Evaluators should avoid an over-attention to model-building and the 
tendency tovard preachment and should not abdicate their responsi- 
bility to sociologists, economists, and systems analysts. The main 
""purpose of many educational evaluation studies will be to increase 
the understanding educators have of particular educational pro- 
grammes. The responsibility evaluators have to be of service to 
educators usually requires a thorough understanding of education as 
a profession, as an art, and as a social service. Few social scien- 
tists can or care to build upon that understanding. 

Basic research on educational issues should continue to be done 
by social scientists as well as by educational researchers. Multi- 
disciplinary studies should continue to be encouraged. However, 
programme evaluation studies, this report assumes, will usually be 
of most value to education officials v/hen developed around the edu- 
cational issues found in the programme rather than arrund the social 
science issues. The advantages and disadvantages of this approach 
should become more apparent in the several sections of this report. 
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Chapter II 

HOW CERTAIN PEOPLE HAVE RESPONDED TO THE NEED 



TO EVALUATE EDUCATIONAL PROGRAMMES 



1, HOW GOVERNMENT OFFICIALS AND 
OTHER PROGRAMMS SPONSORS HAVE RESPONDED 

There are not cany government officials and patrons whu fail 
to sep a need for formal evaluation. It is they who are asking for 
the studies, supporting the necessary budget requests, and express- 
ing an optical sm e.bcut how useful the results will be. They are 
anguished by the decisions that must be made on too little informa- 
tion. They recogniee that limited funds must be split among com- 
peting needs. They plead for evaluation studies to help identify 
programme costs and accomplishments. Jimmy Carter, Governor of the 
State of Georgia, said:(l) 

"We in government are faced with the problem of determining the 
* ideal* level of servicfs within constraints of available 
revenues. ... On what basis and toward what end will these 
programs be directed nhd at what cost? This question can only 
be answered through an evaluation system for social services 
programs , " 

and Henry Ford 11 f the Ford Motor Company said: (2) 

"...the government has no effective mechanism for measuring 
the costs and results of prior legislation agaln.st its 
goals. ... In every decision we must weigh the benefits to 
society against the costs to society, and let the balance dic- 
tate the c^nice, " 

Not all officials and administrat.jrs nre enthusiastic about evalua- 
tion; but in the main they are more positively inclined than even 
the cvaluators. 



1) See Jimmy Carter, "Reply to Position Paoer," Evaluation , 
Spring 1^7A, Speclnl Issue, h-7 » 

2) See Henry Ford TI, "l-eDly to Position Paper," Evaluation , 
Sprin/T 10?/*, Special Issue, h-i;?. 
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Some officials have had favourable experiences with evaluative 
research, but usually the support sponsors give Is based on hope 
rather than experience. Few sponsors have had either good or bad 
exDorlence with formal programme evaluation studies. They become 
painfully aware of how little is known about their programmes, how 
complex the oroblems are, and how elusive the solutions. They are. 
as a group, highly conscientious about the effectiveness of the pro- 
grammes, about doing whnt is best for the constituencies of those 
programmes. Their summons to use rational management methods to im- 
prove those programmes Is energised in bureau after bureau, in 
agency after agency, by benevolent concern and long hours of prep- 
arati on . 

Quite frequently the commitment to evaluate is coupled with a 
commitment to a technological approach for management. Folk wisdom 
and Intuitive decision maKinf;, are found wanting, planning and speci- 
fication are found worthy. The measure of good sense in these com- 
mitments varies from tine to time and from place to place. There 
are many .situations where the various costs of additional technology 
outweloh the benefits. But there also are many situations which 
lack order, preclnlon, and rationality and for which a more techno- 
logical anproach to evaluation Is clearly needed. 

These official.^ in government halls and from other agencies and 
inst<tutions are sensitive to the communication problems mentioned 
in the previous section. They know how difficult it is to find out 
what is goincr on. They know how difficult it is to inform others of 
what is going on. They find the need for documentation broad and 
compelling. They are persuaded that only those things that are 
written down can be shared widely and only those messages that are 
carefully coded can be aggregated to provide a meaningful summation 
across successive years and diverse locations. They support the 
search for documentary evidence of programme success. 

Their reasoning har much to recommend it, but there are grounds 
1-or beinr ca'.tious. Documentation is both costly and simplifying. 
The costs of Keer^in,- useful v.-tord;-. often eat Into operating costs. 
The rerordc k-pt never reflect the full complexity of the programme, 
sometimes providing a simolirtic picture which is easily misinter- 
preted. Administrators occasionally overlook these drawbacks. 
A'-'." the call for evaluation and xhe presence of records, the 

i^ll' f ;v.A thP oreronce itself, are testimony that the admini- 
strators ar^^ doing t.heir ,)ob. In instances where the records are 
not uspful, or ev.r, mir.>ndip.,:. Ih- blame seldom goes to the oerson 
who re.rrr.Hl the^. There is a dan^-er that offlcialr will be more 
concerned about belr... "on recofd" -.s reco^.nising th. n-ed for evalua- 
tion than .b.Mt rnV,<-:r.,- rea:„r. : re.ronro t„ th.-.t n.ed. 



Official ■ statements are made that reflect the assumption that 
what is good for society, that what is good for educational research 
is good for education, and that what is good for the evaluation of a 
particular programme is good for the evaluation of all of education. 
Thus is almost any demand for evaluation supported. These assump- 
tions do not hold. Sometimes efforts made for the one are to the 
detriment of the other. Not all calls for evaluation are going to 
benefit education and society. The officials sometimes respond to 
the need with extravagant claims as to the benefits. 

There is another important effect of the responses officials 
make. They tend to make demands that result in responsibility shift- 
ing increasingly to programm* headquarters and away from the schools 
or projects i:i th»> i". eld. oy authenticating certain issues via 
interview and que»,tionnaire, by emphasizing certain objectives 
through goal statements and test items, or by elevating certain ex- 
pectations and standards with specific definitions in official nomen- 
clature, responsibility for the programme shifts from the people in 
the field to the programme directorate. T;his occurs even where there 
are sustained efforts to use democratic expressions of purpose and to 
use the language of professionals in the field. Sometimes the shift 
is deliberate, sometimes unintended and undetected. Sometimes it is 
good, sometimes bad. But it happens. 

The response of programme sponsors to evaluation needs is both 
well-reason^•d and self-serving. The sponsors have their patrons, 
their audiences. There is a story to be passed along. The sponsors 
want the programme to succeed and they want to be recognised for 
having supported it. The situation is largely the same in a govern- 
ment bureau, the armed forces, a university, or a philanthropic 
foundation. ' 

2. HOW EDUCATORS HAVE RESPONDED 

Teachers, professors, and headmasters have been apprehensive 
about new demands for programme evaluation. They have been con- 
cerned about the success of their programmes, and they have been 
concerned about the increasing demands made upon those programmes. 
They recognise the increased involvement of distant authorities in 
curricular matters, and many are sceptical about the good that may 
come of it. M~; t importantly, educators recognise that any effort 
to evaluate the programme is likely to oerceive inadequately and 
misrepresent at least some of it - and the chances of the programme 
being hurt often appear much greater than the chances of the pro- 
gramme being helped. In other words, there is a fear of formal 
evaluation; and this fear is not entirely without cause. 
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It is not that the^e educators are against evaluation. As a 
group they are quite ready to pass judgement on school affairs, on 
teacher performance, and on administrator performance and to grade 
students on almost everything they do. Not all of them, of course; 
some are quite opposed to course marks. Some are uncritical of al- 
most anything that happens in their world. But most are evaluative. 

Not surprisingly, they see little connection between the kind 
of professional Judgement exercised in these informal evaluations 
and the scaling that goes on in most evali:ation studies. They do 
not deny the subjectivity and limited experience underlying many 
informal decisions but do not s«e that it would be improved by tech- 
nology. They fail to see the re^evnnce to their programmes of most 
of the studies they know which have pla ed a premium on objectivity 
and systemati zation . 

Often educators raise little or no objection to "national" or 
"international" studies. They volunteer their services as test 
administrators. They provide observations of events. They yield 
up their children, often as if that were the most important thing 
the children could be doing at that moment. There is some increasing 
reluctance to be so generous, but getting educator co-operation has 
not been a serious obstacle for evnluators. 

Teacher unions and other professional organisations have started 
tc speak out on the value of evaluation, particularly as it involves 
the evaluation of their members. And, of course, many evaluative 
studies reflect upon the contribution of teachers. The official 
rhetoric of the organisation usually is "pro-evaluation". Their 
oc-:jasional cautionary advice is well-reasoned, and they make an in- 
creasing demand that evaluation responsibility be left in the hands 
of educators. The response of these educator organisations seems to 
be shifting from silent tolerance to occasional participation in 
sponsorship, with some increasing restriction against unpaid service 
by members. 

Many educators would report that they fail to see an increasing 
responsibility for evaluating education. They believe the responsi- 
bility today is not truly different from what it has been before. What 
has happened is increased publicity, increased expectations of edu- 
cation, and increased cost. With more people concerned, there seems 
to be more demand for formal evaluation: but the educators are not 
at all persuaded that formal evaluation should replace informal 
evaluation. 

It is safe to say, thoup:h, that in the. great majority of 
instances prof essiom 1 educntorc will be co-operative with well 
presented and rep.sonnbly modest requests for participation in evalua- 
tion studies of their schools, curricula, and pupil performances. 
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3. HOW RESEARCHERS HAVE RESPONDED 



Educational researchers have responded enthusiastically - by 
and large - to the responsibility for evaluating programmes. They 
are motivated by an honest concern for the effectiveness of education 
and a conviction that rational study of educational processes will 
ultimately improve them. These words of Patricia Story, University 
of Cambridge, reflect that optimism and concern: (l) 

"...an evaluator^s role Is a complex one. He is to help with 
the clarification of the aims of the course, if not with their 
actual formulation; he has a knowledge of sampling techniques ' 
and can organize test programmes; and presumably he must be 
familiar with the content and teaching methods of the course 
he is testing, and preferably have some teaching experience. 
Even if the evnluator is not expected to possess or exercise 
ail these skills, the fact remains that he needs to be a r^^rson 
of protean ability or at least to have very good advisers." 

Hesenrchers have looked ac the schools from many vantage points. 
Many of them question some school ^cals and priorities. Many see 
inefficiency and unproductiveness in the pursuit of worthy goals. 
Underlying their criticism, whether in the laboratory or when giving 
national testimony, is usually ^ deen and enduring ccmmitment to the 
profession. 

The training these educational researchers have had and the 
company they keep reinforce their preference for a ra^ional approach 
to educational management. Most of them are dubious about reliance 
on administrative experience as a basis for governing the schools. 
Many look forward to a day when decisions will be made by objective 
measurement and validate.', formulae. 

Though idealistic these researchers are, they are not immune 
to the tenptation to imply that their own expertise and styles of 
inquiry have been found regularly to be effective aids to policy 
settinfT and routine operations. They recommend looking at problems 
through the eyes of their specialties. An educational psychologist 
can be expected to n;ive priority to measurements of personal talent 
and personality. An ecb.icational sociologist is likely to recommend 
role analysis and the study of community structure. An economist 
emphasizes costs, n historian events. Tt is understandable but not 
a reliable ruide to the concepts nnd rriothods that might be most 
helpful. 

1) See Patricia Story, "Cambridge School Classics Project," in 
Schools Council Research Studies, Evaluation In Curriculum 
Development: Twelve Cnse Studies . London: Macmillan, l^^V ^ , 
36-^*6. 
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And it is to be expected that the .nore the researcher is called 
upon to conduct the evaluation or be a consultant, the more he will 
use it to further his own research programme and to provide training 
opportunities for his students. All very well, except that the 
well-being I'f the programme and the well-being of the researcher do 
not always lie in the same direction. The researcher is to be found, 
as all humans are, under a constellation of motives. Fortunately, 
in circumstances of programme evaluation his motives usually have an 
opportunity to respond in concert,' supporting the use of more formal 
designs in order to learn more about the particular programme and, 
at the same time, to learn more in general about educational pro- 
cesses. 

Most , educational researchers sug.^est a number of ideals for the 
evalu3tioh--design. It should be rational. It should be preplanned 
and prespecified. It should opera tional.--.z:e concepts, not leave them 
abstract and vague. It should give focus to impact on students. It 
should provide feedback for decision making. Few would reject these 
as ideals, but some researchers would set lower standards in order 
to emphasize other design characteristics and to accomolish dif- 
ferent aims. 

The educational researcher has responded to the need for formal 
evaluation of educational programmes by offers of help and with an 
■ expectation that basic methods and concepts, relied on in the past, 
will serve well again. As is his wont, however, he has subdivided 
his methodological cabinet to store new evaluation models and new 
terras. The researcher is showing a growing awareness that the evalua- 
tion concerns of educators and programme sponsors can be a worthy 
challenge to his expertise and can be a justifiable priority for his 
attention . 

A serious problem exists because different neople have such dif- 
ferent expectations of what good evaluation studies will dc^. The 
government official often is lookinc for infor^n.tion thnt will help 
him choose omonr competing programmes and budget requests. The edu- 
cator often .is looking for understanding about a particular teaching 
j.ind learninr: situation. The researcher often is lookln.- for broad 
understanding about tne teaching and learning crocerses. A sin-Tle 
evaluation study, even If suc-ersfully following a good design, will 
not satlsfv nuch different expectations. The purpose of our next 
sect.ion to examine the aims and claims for different rr.ethods of 
eval ua tin>r . 
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THE METHODS OF EVAI.lTATTMr. 



No one method of evalusting educational programmes is suitable 
for all situations. ThP information needs will vary. The audiences 
will have different expectations and standards. The evaluators will 
have different styles, which in turn are more or less useful to dif- 
ferent clients. The purpose of this section is to examine some of 
the differences in existing evaluation methods. 

1. THK MOST COMMON DI>u;NSIOMS FOR CLASSIFYING EVALUATION DESIGNS 

The writing of Michael Scriven has been influential in identi- 
fying basic dimensions of evaluation. His paper "The Methodology of 
Evaluation"(l) identified six dimensions, starting with a distinction 
between the goal of evaluation (to indicate "worth") and the roles 
of evaluation (the different reasons and circumstances for which we 
need to know the "worth"). Blaine Worthen and James Sanders (2) 
created a more elaborate taxonomy of evaluation designs. For this 
report ideas are borrowed from both these sources but presented in 
a simpler and less thorough wav. 

Formative-Summa tive 

The most pervadini- distinction Scriven made was one between 
evaluation studies done during, the development of a programme and 
those done after the programme has been completed. Obviously, a de- 
velopinA progrrimme has components that are completed day by day. It 
is difficult to di3tln.;ulsh between the summatlve evaluation of a 
completed cornponent nnd the formative evaluation of a part of the 
programme. The distinction is not clear-cut. 
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The most useful distinction here may be batween the users of 
the evaluation findings. Elsewhere I have noted that when the cook 
tastes the soup it is formative evaluation and when the guest tastes 
the soup it is summative. The key is not so much when as wh^. What 
is the information for, for further preparation and correction or 
for savouring ^^.nd consumption? Both lead to decision-making, but 
toward different decisions. 

Formal-Informal 

It would be foolish not to recognise the distinction between 
formal and informal studies, even though in this report only formal 
evaluation studies are being considered. Informal evaluation is a 
universal and abiding human act, scarcely separable from thinking 
and feeling. Formal evaluation is more operationalized and open to 
vit:»w, and less personal. It is needed when the results are to be 
communicated elsewhere. Of the two, the formal evaluation study is 
under an obligation to pass tests of accuracy, validity, credibility, 
and utility. 

Case Particular-Generalisation 

A most important distinction is between the study of a pro- 
gramme as a fixed and ultimate target, or the study of a programme 
as a representative of others. Most research is expected to be 
generalised in some ways: over Time, over settings or over subject 
natters, for example. Evaluation research may be done essentially 
to discover the worth of the particular programme, or the worth of 
the general approach. Studies are perceived very differently in 
this regard, both by investigators and their audiences; and a large 
misperception is possibll.e. ■ 

The more the study is expected to be a basis for generalisation 
tne more the need for controls, controlled variation, or careful de- 
scription of uncontrolled variation. Description is needed of the 
changes in tiTne and place nnd persons, and in many of the ways in 
which generalisation may be directed. The case study undertaken for 
either knowledge of the particular or for generalisation is a more 
useful document when It provides the reader with a vivid portrayal 
of the setting and context of the teaching and learning. 

Product-Process 

Another dimension on which evaluation studies vary is as to 
whether they give primary nttention to the outcomes of the programme 
or to its transactions. A r,tudy of th^^ "product" is expected to in- 
dicate the payoff value; a study of the "process" Is expected to 
indicate the intrinsic values of the prof^ramTie. Both are needed in 
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any effort to get at a full indication of the worth of the programme, 
but in any actual rtudy only a small portion of either can be exam- 
ined. Much of the argument as to preferable methods depends on the 
bjeliefs held as to which :.s more mensurable and useful. 

Descrip tive-Jucir^e:: 1 1 : 

Many evfiluators comin,^ t'ro^n a T.ocinl. science background define 
the evaluation task larvely as one of providin/!; information, with 
an emphasis on objective data and n de-emphasls on subjective data. 
Those coming from the humanities are likely to reverise the emphases. 
One will find .some r.tudies hir;hly descriptive of students and set- 
tings, providing- careful repo^^ts of differences and correlations, 
but with little direct reference to criteria of worth and value 
standards. And eisevhere one will find evaluntion studies nrobing 
into the pluralism of values to.be found in c-ny educational setting-. 
As with any of these dimensions, any particular study is not likely 
to be at one pole or the other, but to make some combination the 
comproraise. The extremes identify a dimension on which some varia- 
tion if ippni'ent from r.tudy to study. 

t'reord ina te-:- e.''rions i ve 

Studies differ cons i dernbly as to how much the issues of evalua- 
tion h-e determined by observation of activities and by realisation 
of concerr.s ^f p^rtic i oa:i ts in the Dro^-rnmrne. Preordinate r.tudies 
are more --riontf-d to objectives, hyootheses and nrior expectc.tions, 
mediated by the abstractions of lnn-ua*:e. ir-eord inn te evnluators 
know w:-,at tney nre lonki:,,- for and desirn the study so as to find 
it. .•••-•L-:ii-r!;:: ve studiMS n re orr-:nlsed n round phenomena encountered 
- otter. unexrjK.'c ted ly - ,-^s the pr^^ r^rrimmo ^^o^s alon^:. (There are 
ways of being prespecifled and responsive other than these, of 
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In a -tr-.-r^rvt i r;;: t-j' :::i:dv a re:aM-/e:v lar^-f or.rtion of resources 
is s-ont on -ettin.' objectives so^-cifiod in writirp: .md develooin^r 
instrM:n-n:...: -r.i r----:, ; in nr^vld^r-' for or controlling: vnria- 
ticn t^ v:-:.-! --M^e ^-o^nda b !.p si -i teme:; ts of r-elationshin amonr 
vrji'i . ^ ^ rosp;)risi vo studv a rel a t, i ve Iv iar^e portion of re- 

.souro^^n is sr-nn: in or'-rvirin;* and olacin.' observers on the scene. 

^ho ! 1 s L ■ -^-Ana v t i r: 



f'^r airi a:-, t- now m-jch ^hey treat the prof'r*a;nme as 
a totality, roi-.rn i !■ i n.' -micop tj^-^ ; unri r i fs common to nor)-t echnica ] 
ai:d : i;r:rt':: . .h- rr ^t-- < ojn::r:;n : a 1-r ■ ; enr.e r*p:;parch aporoach \r to 



concen*.i*a tt - ar-. 



"hamc tor* i t i c s . A cascr s tudv 



is often to preserve the complexity of the programme as a whole, 

whereas a multiv«rinte annlysis is nore likely to indicate the re- 
lationships a.nong descriptive variables. 

Internal -Externa I 

An obviously important difference in evaluation studies is.- 
whether they will be conducted by perr.onnel of the institution re- 
sponsible for the programme or by cutsiders. They differ as to how 
formal the agreement to evaluate, as to how free the eval.uators are 
to raise issues p.nd interpret r indin.■'^:. , nrui as to how changes ^■ 
plans will be negotiated. 

The eif-rht dimensions above do not result in different 
evaluation designs. V.any of tht dimensions are ct^rrela te'i , both 
conceptually and in f requency-of-use . For example, an "internal" 
evaluation i;';udv is more likely to be formative than s\:mmative, more 
likelv to be le5-critrr!v< than .jijdirmental. These characteristics and 
correlations mirht be particular to the place^^ where evaluation has 
been most common. In new evaluation r.i.tuntions the key dlmensionis 
and combination/; r.:.-ht qi.i; te different. 



Anothei* way to look at the different ways educational pro- 
grammes are evaluated Is to look at typical approaches. The dif- 
ferencef. between the aoproaches can partly be described in terms of 
the dimerir.icn.s .ju^5t discussed, but more subtler characteristics become 
apparent when mod-jls or prototypes are examined. 

Ir ti'.lr. :',ecti:)n» nine -aDpro^^chcs will be considered. The first 
two are very common. Tn any one year at least 10 per cent of 
Amerlcpn teachers and puni'-s are involved, at least for a few minutes, 
in student achievem^•nt testing '-^^ Institutional self-study, as part 
of a forr.ol evaluation effort. Some of the others are typical as 
part of informal evaluation, but uncommon as formal studies. The 
last two are much more rare, but incre.^^sin.'l y mentioned by evaluation 
consultants. l-or the deener unders tand i r^r of this chapter particu- 
larly, it is advisr-.ble that it is read in oon.iunction with the CECD 
docuirent "Case litudips in the i-valuation ot: Kducational Programmes" 
which is available free on der:^nd from CKiH / OK'JD. references in the 
text to case studies are to those i :: that dnni^rnen t . 

Student Gain by Testir).- 

The approach usually suf^^-^psted ly measurem^rit sdoc i a 1 i str, and 
educational psychologists i "testing to measure student rain in 
performance". Tt relies on tests developpd to match prespeci f icati on 
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of objectives or on standardized tests that match or cover programmf 
objectives (or their correlates). Many studies are undertaker, using 
tests developed primarily lor counselling and guidance purposes. 
Educators in fav.mr of highly structured curricula prefer criterion- 
referenced tests as a basis for measuring student gain or mastery of 
the task. Control groups are sometimes used. Analysis of regres- 
sion or covariance is sometimes used to identify var.lance in student 
scores attributable only to the teaching. The method is found weak 
by some critics because the tests under-represent what education 
apparently does for school children and because the identification 
of poor learning is often not much help in identifying or correcting 
deficiencies in teaching. 

Two of the case studies available in the companion document 
are primarily student-testing based, the Michigan Assessment study 
and the evaluation of "Follow Through". (However there is some stu- 
dent testing in most of the case studies here.) Both of them were 
set up to Improve programme management and to provide information 
for stare and national policy setting. Neither evaluation study was 
greatly successful in these roi^s. A third case study, that which 
resulted in the Coleman Report, used a great amount of student test- 
ing. Partly l^ecause of action by the U.S. courts the Coleman study 
more effectively sustained its readers' concern about government 
policy for schools, desegrep;ation, and educational opportunity. Most 
of the "accountability laws" passed by state legislatures in the U.S. 
imply cr m-ndate a student-testing approach.. 

Ir.sritutiona i, Self-Study by Staff 

Mostly because schools in Amori'ca have, by law,, a great deal of 
autonomy, their officers have lookt-d for Wciys of evaluating them, to 
avoid federal and state control. Long bc-q they formed regionjil 
alliances, a network of schools, for the purpose of accrediting eac:i 
other. A number of sneclal professional organisatlon.s ,, sucn ac- th- 
American Association of Vt-dical CollGffes and the :;aT:jnnal Council for 
Zhe Accreditation of ^eacher ^:ducation, moved in the £fame way to pro- 
vide self regulation and to avoid state ret'iuln ticrn . 

i'he -: :ncipru method of evaluation adopted by these organisa- 
tions ha., been facultv self-studv . This is not to say that the final 
word on any matter was left to the school »s faculty, but the primary 
^-:athering of 'Inta, in^'erpre tin^^ of problems and recommending improve- 
ments has beer via the initiative and hard work of the faculty. 
Review by visiti-^- committees and ndherence to specifications adopted 
by member school,^ .^ns also been common to the seif-regulation process. 

The self-study i sometimes uried by an Institution under inter- 
nal press-ire. without r>r.y externnl requirement, such as when students 
are protesting or when major bud^-^ret ch.qnros are imminent. Tt is a 
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procedure which honours the status quo , establishmentarian values. 
It takos a heav7 to:i 1 in staff time, and is a subject in contract 
negotiations with some unions. It has the great value of keeping 
problem-solving responsibility at the site of the problem. None of 
the case studies gathered here illustrates this approach to pro- 
gramme eval uati on . 

Prestige Panel or Blue-R.ibbon Panel 

A third common evaluation approach used by governments and 
organisations of all kinds is the panel of leading citiz.ens, usually 
people without expertise in education (or whatever the focus is), 
but who are held in hi;h esteem, who have a strong sense of social 
responsibility, and whi- are respected for outstanding achievement 
of some kind. A group :)f several such people is asked to study a 
problem. They may follow their intuitions, or be guided by an ex- 
perienced cuunsel or staff member. They are expected to make a very 
formal report, usually in a matter of weeks or months. 

A W9ll-knov,Ti example of this approach is the British report on 
"children and their primary schools" prepared by Lady Beatrice Plowden 
and her comI^it^ee . ( 1 ) In some countries suoh studies are often 
used as grouiids for the enactment of corrective legis:^Gtlon. The 
blue-ribbon panel, also, is often the first choice airio-.;., rvaluation 
methods when matters are "in extremis" - seized upon by leaders 
when tne institution has been greatly injured, or is immobilized by 
crisis. 

Less sensational instances are common in the schools when a new 
curric^ilujn or student policy or staff organisation is needed. A 
prestigious group is asked to investigate. It almost always is ex- 
pected to make recommendations. In these -instances the method be- 
comes similar to the self-study approach menti *?d previously. It 
should be noted that the distinction of the panel members permits 
them to use personal experience and Judgment as an adjunct to and 
sonetiraes in lieu of more objective e^nd definitive data. 

none of the crise studies is an example of this approach. It 
might be noted that, in the case of the N.-' chigan Assessment Programme, 
the Michigan Educational Association, s professional organisation for 
teachers, sponsored a study of the state assessr.r^nt by engaging a 
blue-ribbon panel of evaluation specialists, (2) 



1) See Ladv Beatrice Plowden et al. , "A Report of the Central 
Advisory Committee on Children and Their Primary Schools," London: 
Her Ma,l'esty*s Stationery Office, 1967. 

2) See Ernest House, Wendall Hivers, and Daniel Stufflebeam, 
"Assessment of Michigan's Accountability System". Michigan 
Educational Association: Teacher's Voice Supplement , April 22, 
197^. 
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Transactlon>>O b servat.lon 

In contrast to the student-testing approach as a (sometimes) 
ditxiplined study focusing mainly on educational outcomes, the 
transaction observation approach is a (sometimes) disciplined study 
of educational processes. Here the activities of the programme are 
studied, and with special attention to settings or milieu. Issues 
are often drawn from the proceedings rather than from theory or from 
gual stat-^ments . 

Disciplines fro.ii which the methods of observation come include 
anthroDology, ethno^!;raphy , history and journalism. Some of these 
disciolines emphasize the crucial importance of reporting as well as 
of mer.iiuring. This evaluation approach follows that lead. 

The transactions emphasi^.ed are not those between evaluator and 
educator, although it is the evaluator with this approach who seems 
to be especially sensitive to orofessional role relationships. This 
is nicely lUustr^^ted in the UNCAL case study. The transactions 
emphasl-.e-.i aro those that take place in the classroom as learner 
encounters instructional arrangements, plus the many transactions 
air.oni:: educators makln,^: those arrangements. Observations are made . 
in -global fashion, either by trained observers or by participants 
who are debriefed by trained interviewers. 

The SIA evaluation is an especially good example of this 
approach. Several Swedish schools had comoleted an experimental 
teaching pro^^rnmrr.e . The evaluation was an ad hoc study to see what 
had been learne.-i. The r^rincipal attention was to what the teachers 
had dene, with oartiruln.r reference to how much the instruction was 
influenced by institutional and cultur.:-il frame factors. The highly 
political nef'oilations to initiate the evaluation were an instruc- 
tive aspect 01' the study. 

In the transaction-observation a-ip roach one often will find an 
actenrion to the niuralisrii of values in education. This is ccnsis- 
trnt with th^^ Pier, -ade by Fran.rois Hetrr^nn in a 1073 OHCD document .( 1 ) 
fie first extrnctor: a sentence from a Heport of the (U.3. ) National 
Academy of Sciences (' ■-'O: 

"Ahatevor i'nprovement mi,^:ht be made in assessment systems, 
l.-.eveforo. it is important to remember that the products of such 
s> stems ^Jltlmnt.-;.y represent no more than Inputs into the com- 
plex netvoTK of decis[on-makin>7; processes, nrivate and publ^ic , 
er^cnomic r.^nd political, that tof:^ether mold the growth of tech- 
noIo/;y and channel its intef?ration into the social structure. " 



1} See Franco is ::etman, S ociety and the Assessment of TechnolofT v, 
Paris: : ^r. :ani sa 1 1 on lor Economic Co-operntlcn and i)evel ODi.nont , 
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"But is this duty to fall only and exclusively to the centnU 
decision-rjaking authorities? In a pluralistic society, sr,h a 
monopoly in formulating and assessing alternative future^ ootionj 
and courses of action may be regarded as a contradiction If not 
as a real danger to democracy. Hence the idea of a "corccenti ve 
assessment", in other words, of such an institutional spt-up r.3 
would allow different sectors of activity, organisations, groups 
of interest and affected parties to make assessments of their 
own, on the basis of their proper r.tandpoint and scale of 
values." 

It is not unusual to find populistic sentiment in the uransaction- 
observation metncdology. 

Instructional Research 

Vhen many educational researchers are nsked to recommend an 
evaluation approach they speak of an experimental design, with com^ 
parison cf randomized treatments under controlled conditions, or as 
close an approximation as pos.'.ible. They urge the investigator not 
to pass up the chance to contribute to the general knowledge about 
teaching and learning, sometimes uaying little attention to whether 
or not the study is useful to the people involved in the programme. 

Some authorities believe that the exoerimental approach is 
essential even to an understanding of the particular programme. 

"...the small differences [educational programs] are likely to 
make can easily be either overestimated or missed entirely by 
comparing the treatment group with a non-comparable group. The 
only truly satisfactory way of dealing with this problem, of 
course, is through randomly assigned treatment and control 
gx'^oups . " ( 1 ) 

There is a great respectability to this approach, but it has 
come increasingly under criticism for its poor record of assistance • 
to practitioners and policy setters. In a section (Responses of 
Administrators to N'ine Key Questions) further on. John Misbet is 
quoted reflecting concern about the educational research aooroach 
and the student-testing approach, esoecially in formative evaluation. 

But the purpose of evaluation sometimes is to spe^k of summative 
values and to provide generalised information. Such was the field 
study of programmed biology materials both prepared and evaluated ty 
Richard An.ierson. and reoorted as a case study in this book. 



1) From John V. Svans, "Evaluating Kducational Programs - Are We 
Getting Anyi-vhere?". Educational Researche r. Vol. 3t No. ■, 
September 1^7^» o. 9. 
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Robert F. Boruch has published an excellent bibliography (l) 
oi' particular studies and general guidelines on the evaluation of 
randomized experimental programmes. His references include pro- 
grammes of social rehabilitation, law enforcement, socio-medical 
experimentation and welfare as well as education. 

Managemen t Analysis and Social Policy Analysis 

The next two ?.7»proaches blend into each other but represent 
differences in the urgency of the findings and a'ldiences. When an 
evaluative study is done to assist programme managers (to make im- 
mediate or repetitive monitoring decisions) the approach might be 
called Management A:^alysis . When the same study is done to assist 
policy-making, perhaps with a longer time-span, perhaps for a wide 
constituency or for governing board members (rather than managers) 
the approach might be called Social Policy Analysis . But th.^y are 
similar In many i-espects. They both drew upon the social sciences 
not usually Involved in educational reoear'ch, manag^^raent perhaps 
more often raising economic Issues, policy-setters perhaps more 
often raising socio-political issues. They both (iraw up such econo- 
mic concepts as cost-benefit analysis and productivity coefficients, 
and such sociological concepts as opportunity costs and work ethic. 

One case study reported here which Illustrates a social survey 
approach to policy setting is that of the Coleman study of "equal 
-opportunity". The principal data are achievement test results :)Ut 
the style of analysis and the interpretations are those of the social 
policy analyst. 

The evaluation cf the Illinois Program for Gifted Children and 
of Ottawa's Informational Retrieval Television are both studies 
undertaken by the sponsors of the programs.-? (one p^ovemment, one 
private) to help decide whether the prori^^^-nmes should be expended, 
cut back, or changed in some ways. In botr; cases the officials, 
finding that no startling facts were likely to emerge, took action 
before the several years of findings were available to them. 

Goal-Free Evaluation and Adversary Evaluation 

These two approaches are newcomers to the educational research 
scene. Michael Scrlven Introduced goal-free evaluation with t^.e 
conversation rooroduced in the next chapter. To avoid co-option he 
stressed keeping distance between evaluator and orogramme staff, 
ev-^-' to the extent of not knowing what the J^taff goals were. 



1) See Robert F. Boruch, "Bibliography: Illustrative Randomized 
Field Experiments for Program Planning and Evaluation". 
Evaluation , Volume 2, No. 1, 197A, S3-88. 
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Scriven -s evalustor Is aware of what goals are usually pursued and 
is supposed to be sensitive to a groat range of indic.Uor.s tnat 
attainments were made, so the approach is not goal-free .n that 
sense. A highly structured checklist of evidences is utilised. 

The Adv--nary approach has several champions, most prominently 
Murray Levine. Thomas Owens and Robert Wolf . ( 0 The resources for 
evaluation are divided in two: part to show the shortcomings of the 
programme, the rest to show merit. In some cas^^s the court of law 
is taken as the moHeT . with the testimony sh iped. the case made, and 
with cross-examination by counsels for the prooecution and the de- 
fence The approach has an unusual command of the use of real time 
for decision-making, an asset that few other approaches can match. 

These nine prototypes are over-simplifications of the approaches 
evaluators actually use. Most actual studies draw upon several 
styles, varying a:, the programme, the issues and the audiences 
change. 

on the following page, a grid summarises the features of these 
nine approaches. (A bil.liography of readings relevant to this vill 
be found in the Appendix.) 

Llice -ivangelist preachers and high pressure salesmen, evalua- 
tors promote their methods. There is good and bnd in each of them. 
The task for the consumer is to pick the method for which the things 
it does well are important and for which the things it does poorly 
are unimportant. But there is all too little agreement, all too 
little evidence. oS to what each method does accomplish under which 
circumstances. 



3. THE COSTS OF SVAUJATIMG 

An evaluation study seems to cost whatever the funding agency 
can afi:-:J. The exoerience of many evaluators indicates that the 
director will emoluy as many people as the funds will allow, once 
the costs of instruments, data processing, travel and administration 
are -aken care of. A good estimate is that 30 per cent of evalua- 
tion budgets go to professional salaries. 

On« common procedure of. funding agencies is to set aside a cer- 
tain portion of the programme's operating costs for evaluation. For 

1) See Murray Levine. "Scientific Method and the Adversary Model: 
1) See ^^^^y^^ii^i^'ry Suggestions". Evaluation Comment. 19^?. 

Thnl.nq^n'" Owens. "Educational Evaluation by Adversary 
proceedings"! !A E^^est House (Ed.). Sc_hool Evaluation: The 
Pn??tlcsfnd Process. Berkeley: KcCuTcnan Publishing 

Ro^rt' ' '"Solf^-ThrApolication of Select Legal Concepts to 
Educational Evaluation." unpublished PhD Dissertation. 
University of Illinois. 197^. 
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large programmes using conventional methods summative evaluation 
tnight run up to i' pf:^r cM?nt. Smaller programmes, unusual programmes, 
and highly controversial pro/,:rammes may need b per cent. Formative 
evaluation can be more expensive, dependin/:: on the difficulty of the 
development problems. In evaluation efforts of the U.S. Experimental 
Schools it became apparent that nt^oncit-vs could not effectively use 
substantially more than ^ per cent, perhaps because expectations as 
to what would be accorr.pli shed by the evaluators becomes unrealistic. 
(The figures in this section are based on the author':> own experience 
as a project director and consultant.) 

One good strategy for the commissloninr ardency is to be explicit 
as to what kind of observations or what Xinri of information It needs, 
and to get the prospective evaluators to indicate how they would do 
it and what it would cost. They might be encouraged to submit more 
than one olan, with coF.ts estimated for each. Experienced evalua- 
tors will be able to give examoles of what other studies have cost 
- still it is difficult for the commissioners to knov how much was 
accomplished for these expenditures. 

The 'duality of the study is not likely to be related to its 
cost. :'he princinal concern Is to get able persons working on the 
studv. Sometimes r/ne\ cort very little, sometimes .they are not 
available at any nrice. Ihe commissioners have to pay the "going 
rate" for what i:hey want, iust a? they would for architects, medical 
doctors or string' nUs^trtets. 

The costs of an evaluatjon study are not only the exnenditure 
of financial resources, but also largely the use of stiK-^mt, teacher 
and administrator tu:ie. Interference in the programme operations is 
a substantial ccst in some studies, sometimes even causing the 
evaluation results t** be called into question, and encouraging the 
evaluator to seek unobtrusive methods. (1) A drop in staff morale, 
community unrest, and increased bureaucratic insensl tivi ty are in- 
tangiY>Ie costs tha r are sometimes attributed to evaluation pro- 
grammes. (^) The tot^: costs of an evaluation study are extremely 
difficult tc tally. 

There are some reliable expectations as to how financial costs 
will mount, in addition to the obvious fact that employing mere 
people costs moi'e monev. ;!ere are a few: 

1) See Eu,frene J. V.>bb et al., unobtruslyp Measures : uonreactlve 
hesear'ch in the occial .Sciences . Chicago: iTand :'ici;ai^.y 
Company , ^ . 

2) See ir.rnest . ;:ouse ' Kd . ) , .'chooi l^vnluation : The Politics and 
Process. l-'O r 1 o : r i tcrian Publishing Coro era t1 on , 19V;'j. 
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1. Sa^apling a large population is much ?.ezz expensive than 
interviewing every teacher or testing every child. How- 
ever, with small populations, such as that of a single dis- 
trict, the administrative co3ts of sampling can overcome 
the savings. 

2. The size of sample by specialists in sampling increase 
cort? of data gathering disproportionately above the size 
needed i^r answering most evaluation questions, 

3. The cost o,"' constructing nnd validating new instruments or 
procedures i very high. Existing ones will usually be 
much cheaper, although perhaps not sufficiently relevant. 

4. It is usually a waste of resources to use a standardized 
instrument for a small number of observations if the 
instrument is not a part of the clinical tools of the eval- 
uator, t^e educators or the audience members. Unstandard- 
i^^-a observations or questioning are likely to be more 
informative and less expensive for small operations. 

5. The use of computers is necess?i^ in most large-scale data 
analyses. Computers can save groat amour.' r^s of personnel 
time, but their budget lines are often over- funded. De- 
signs for evaluation studies are consistently overly ana- 
lytic because the computers are so readily available, the 
costs so easy to Justify, and the results so apparently 
credible. In small studies especially , computer costs 
should be considered as needing thorough .justification. 

6. Publication and dissemination costs are regularlv much 
higher than anticipated if there is to be an ef:':ective job 
of circulating the findings. 

7. Overbooc^ costs to institutions for use of facilities, li- 
braries, access to experienced personnel, fringe benefits 
for employees, etc., are as justifiable for evaluation 
studies as for more conventional research studies. 

It would \e nice if we could catalogue evaluation costs, showing 
what findings can be expected to cost, or even what inquiry methods 
are relatively more costly. But no reliable information of this 
source can be provided. The commissioner of an evaluation study 
will have to question experienced researchers and examine budget in- 
formation from completed studies - and still should expect to rely 
on guess-work. As with many commodities the prices are very unstable. 
But given reputable evaluation personnel and with care in writing a 
good agreement, the commissioner can get his money ^ s worth. 
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Chapter IV 

NEGOTIATION OF AGREEMENTS TO DO EVALUATION STIHDIES 



The general purpose of this report is to review the various 
methods of programmo evaluation currently in use in education. A 
nore specific target is the review of the particular responsibility 
for negotiating an agreement. to carry but an evaluation study. In 
this section the reader will find a listing of issues needing con- 
sideration by the commissioners or sponsors of the study* and by the 
prospective evaluators. Three hypothetical co.iversations between a 
commissioner and a prospective evaluator have been included to sug- 
gest different dir-ections initial conversations may go - depending 
on the orientation of the evaluation specialist. Comments by ex- 
perts considering the difficult task of negotiating an agreement to 
carry out an evaluation study are also included in this section. 



1 . ISSUES FOR CONSIDERATION 

It is reasonable to suppose that in addition to the usual con- 
sideration of purposes, dates, persons, and costs, the parties 
working toward an agreement would anticipate the things that could 
go wrong and take preventive steps, and that they would make real 
preparations to deal expeditiously with the problems that will occur. 
That is the pessimistic supposition taken here. "What could go 
wrong will go wrong" is a useful slogan. The following advice about 
issues is based not only on theory but on the practice of evaluation 
in many places, as illustrated in the several case studies in the 
final section of this report. 

A contract can be too specific or^ot specific enough. If the 
contract is too specific the evaluator will be unable to respond to 
unexpected difficulties and opportunities. If the contract is not 
specific enoup:h the chances of the sponsors getting what they need 
(and having the ,-rounds for insisting upon it) are reduced. The 
conversations between commissioners and potential evaluators can al- 
so be too specific or not specific enough. If they undertakt a de- 
tailed technical examination of the programme, the evaluation design, 
and the possible troubles that mirht occur, the conversations are 
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likely to de-emphasize the larger policy questions, to disdain the 
more subtle "unmeasurable" circumstances, and the parties are likely 
to over-protect themselves with constraints and risk-lowering clauses 
that could severely limit the opportunity of the evaluation study 
to be maximally useful. If the conversations are not specific enough" 
(and this is usually the case, sometimes because the commissioner 
does not want to reveal how little he known about research methodol- 
ogy and the evaluator does not want to reveal how little he knows 
about education) then there is little chance for tho evaluation 
study to engage the questions the commissioners need help with and 
to extend the understandings of the programme in ways that will 
facilitate decision-making. 

Purpose 

Perhaps the most important question to be raised, once and 
again and again, is that of the purposes of the evaluation study. 
Of course, the purpose of evaluation is to find out what is good 
and what is bad. In programme evaluation the purpose is to find 
out the merit and shortcoming of the pro>:ramme. l-ut the particular 
questions that concern participants, sponsor?, and other constituen- 
cies, need more final judgments. They need information, inter- 
pretations, recommendations. Tr.'^y need attention to the particulars 
of the programme. A general review of the merit and shortcoming may 
fail to address these questions. A hypothesis-testing experimental 
study may fail to ad.Jress these que.itions. Any design ^ might. There 
may or may not need to be direct attention, for example, to the 
actual achievements of students. Questions of whether or not the 
matei'ials are too difficult to read, aimed at the wrong objectives, 
offensive to parents; questions of whether the teachers are too 
liberal or conservative, whether or not they know their subject 
matter; questions of the competence of programme administrators 
- thoso are Ei few of the many imoonderables that may be on the 
minds of programr.:-.' constituents. The evaluators and commissioners 
need to examine the range of pertinent questions and to allocate 
the usually modest resources for evaluation to the few questions 
that can be given primr.ry at**ention. Evaluators and commissioners 
need to consider - not .so much the dictionary definition of evalua- 
tion, nor the designs of the evaluation methodologist - but what 
the people involved expect an eva],untion study to accomplish. 

This is not t..) say that a rood written statement of the pur- 
poses or expect.-.-* ti.onc of the evaluation study is not necessary, or 
that it will assure that the direction of the evaluation is proper. 
The main thinly is for the nartlos to be satisfied that they know as 
much as chey shoi^ld about why the evaluation study is being under- 
taken. 
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Audiences 

It might at i'irst be assumed that the commissioners are the 
primary audience for the evaluation study, but there usually are 
others to whom the results will be circulated. Quite often, the 
commissioners do not expect to learn snything new, and expect that 
the study will but confirm what they already know. Evaluators need 
to know who the audiences are so that they may gather data consid- 
ered most relevant by these audiences, verify observations to meet 
their standards of credibility, and prepare a presentation of find- 
ings in a style that the audiences will find comprehensible and 
useful . 

It is obvious that parents, ter«chers, students, officials, pro- 
fessors, and others will differ, and diff.-r within their own groups, 
as to vhst isrues In the T>rof:ramme are most important. Nost may be 
concerned with whether the nrogramme is accomplishing its instruc- 
tional ob,-)ectives, but some will not. Some of the time, energy, 
and funding for the study may need to be spent studying audiences. 
How tht^ c'.^mmis.3i:.ners perceive their audiences is nn important first 
step, and one th.nt needs to be re-checked durin^^ the progress of 
the study. 

The evaluators have their audiences too. They want their find- 
ings to be shared by research colleagues, they want their good work 
to be recoF;ni:ied by their superiors, and they want their students or 
apprentices to learn from the case at hand. Some recognition should 
be made by the commissioners of these oudiences. 

Methods of Inq uiry 

Both parties to the nerotiations will have some idea what is 
appropriate and inppr^ropria to evaluntion methodolo.r;y for the study 
at hand. There needs to be sufficient discussion of how the study 
might be carriod out so that neither party will feel misled about 
what is expected by the other. The lifting of the nine approaches 
in the previous section might be used to illustrate some of the 
techniques desired by either party. If at all possible the pros- 
pecxive evaluator should provide representations of previous v/ork 
(perhaps a portfolio of instruments, protocols, and reports.'. The 
commissioner might show examples of r:cod and Door studies previously 
done to help the evaluator understand his standards. 

It is reasonable to sunpose that the evaluator vi U be the bet- 
ter judge ap. to which techniaue is most suitable "or answering a 
particular question, at a particular cost, and at a particular level 
of credibility. But it should also be reco/^nised that the commis- 
sioner ^ill have important ideas as to which techniques have been 
useful and which have been ob.iectionable in other ciM..jn:-tances . 
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Because the commissioners feel at a loss, sometimes, to know whether 
or not a method is reputable, it is not unreasonable for them to 
appoint a consultant (or for the evaluator to employ an independent 
reviewer) who will attest to the suitability of methods. 

One of the most surprising things to .commissioners is the fact 
t.'at distinguished researchers cannot or will not switch their 
methods of inquiry. When one chooses an evaluator, one chooses a 
method of evaluating. Investigators will often redirect their at- 
tention to emerging issues and to new opportunities to gather infoi- 
mation, but they are less flexible about changing styles of data 
gathering and analysis. One way they increase their range of methods 
is to add persons with different styles to the team, or to sub- 
contract a part of the study to another group. 

An important part of the methodology is the reporting technique. 
Evaluators have many different styles of reporting. Experimental 
psychologists, philosophers, and many "academics" as a group prefer 
to write a report suitable in style for publication in a scholarly 
Journal. The report mcy be prestigious, but may be incomprehensible 
to some important audiences. Some commercial evaluation agencies 
and local study-groups are prone to make reports that reflect most 
favourably upon the sponsor. These have public relations value, but 
are a poor aid to decision-making. Whatever proposal the prospective 
evaluators make should indicate the style and extent of reporting 
expected. The commissioners should review this potential commitment 
carefully . 

Confidentiality 

Many data used in evaluation studies require confidential hand- 
ling. Official correspondence, teacher-competence ratings, data on 
parental income, student reports of misconduct, are some of the more 
sensitive data, but many individual observations and scores are 
.gathered with the underr,t3nd.ing (often implicit) that they will dis- 
appear into impersonal summary statistics, their identity remaining 
undisclosed outcide the boundaries of tlie evaluation project. It 
used to be that burying information in a computer or in a student's 
"cumulative file" was considered secure storage, but increased ac- 
cess to computers and files, particularly with new legislation pro- 
tecting the privacy of individuals, has changed things. The plan 
for data storare and procerus! n:" anrl for circulation of interpreta- 
tions should be a major provision in the evaluation-study plan. 

Certain rules about the release of scores, observations, and 
quotations will sometimes be needed. It should be stated explicitly 
which people will have =;ccess to the data, the access an individual 
will have to his own records, how errors nay b" recognised and cor- 
rections made, etc. Any transcripts of tape recordings of meetings. 



classroom sessions, interviews, etc., may n^ed to be cleared by all 
who participated. Setting such rules is an integral part of the 
technology of educational measurement and is part of the ethical re- 
sponsibility of an evaluator. 

One of the most common disagreements that arises between spon- 
sors and evaluators is with the reporting of programme information 
considered confidential. When this occurs it is likely to be be- 
cause the evaluator felt obligated to look into matters that were 
not originally anticipated, then, holding data the programme people 
recognise as relevant but not appropriate for all audiences, he in- 
cluded it as a report that becomes available to those audiences. If 
the findings are extremely bad the evaluator may feel a mora"" ob- 
ligation to act unilaterally and release them even though -^he origi- 
nal agreement specified that reports would not be made wit: >ut the 
permission of th^ sponsor or programme personnel. Such situations 
cannot be dealt with specifically in advance, but it is wise for 
the commissioner and evaluator to be aware of these possibilities. 

Continuing Negotiations 

Even though a formal contract or agreement is signed it is im- 
portant to consider the arrangements between cormnissi oners and eval- 
uators as requiring monitoring and further negotiating. It often 
is a mistake to fix a priority or a limit to the study when a bet- 
ter decision can be made after more is understood about the programme 
and its evaluation.. The need for such continuing negotiations should 
be anticipated and provisions made for modifying the original arrange- 
ments. 

There are bo^ond to be misunderstandings such as the one mentioned 
two paragraphs back. How to deal with misunderstanding is no great 
mystery - one needs contact, comJDunication , empathy, and a willing- 
ness t:o reconsider. But an impasse sometimes is reached. If so, 
it might have been helpful to have identified, in advance, a group 
that would act as a ''court of appeals". The original negotiations 
should anticipate the need for subsequent negotiations - not as a 
sign of fallibility, but as a sign of responsibility. 

2. chhx:klist 

To assist both the commissioner nnd the prospective evaluator 
in covering the many important considerations, the following check- 
list has been prepared. 
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Checklist for Nerotlatlnr an Agree ment to Evaluate a n EducAtinnm 
Programme 

Do the parties to this ne?^otiation know each other? \vhat 
more dc ney need to find out? Who wants an evaluation 
study? Would those not participating (e.g., programme de- 
velopers, teachers, students) have added an important per- 
spective? 

What programme is it that is to be evaluated? ;^ose pro- 
gramme is it? Whcit is its setting? its history? its pur- 
poses? its scope? How has it been evaluated before? 
Why is there to be an evaluation study? What is it expec- 
ted to produce? What should it accomplish (e.g., recom- 
mendations, authoritative Judgments, explanations, points 
of view) ? 

Who are the audiences for the evaluation findings? Will 
different audiences (e.g., parents, technologists, members 
of parliament) have different background experience and dif- 
ferent Information needs? 

What do the people who are most closely involved with the 
pro^Tramme see as Its ma.lor issues or problems? What issues 
do other people see? How do all these relate to the major 
issues facing education elsewhere? 

What resources are available for the conduct of this study? 
What cost estimcites can be :nade (e.g., in money, staff time, 

p:*o>^ramme di ^j'^uption) ? 

What is the work history and working style of the prospec- 
tive ev-luators (i.e., the persons, ream, or a/^ency)? X)o 
they have a portfolio of reports and artifacts from com- 
pleted studies? 

V.hy woi:ld tho -VcVmi'> torn be interested in doing this study? 
'.v'hat is ther- Ir. it for them? Who else would they like to 
hRv,? he In in- thorn with it? 

What will be the primary sources of data? V/ha t arrange- 
mentj; would be necessary to gain access to these sources? 
A r o r i. 1 1 e s o f a c c o s s needed? 

Purinr: the :^■^■■^r^^>- ■ f U\e ^?vaLuatlon study, where and how 
would th^> riotn oe l^-ept-? What would be the rules of access 
to thei'.o -'Ptp (o.r,, to partirioants, sponsors, newspaper 
reuor' isorr- ) ? 
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11, What would be a suitable plan for reporting the findinrc^s? 
informal feodback? progress reports? final presentations? 
Are the evaluators free to publish findin^i^ in professional 
Journals? 'fihat checks will be made on the effectiveness 

of the evaluation feedback? 

12. How will further arrangements be negotiated after the study 
begins? What will bo the response to unexpected chancres in 
programme? What misunderstandings may arise between the 
sponsors of the study and the evaluators? How will con- 
flict be resolved? 

)3, What more needs to be said about the purposes and expecta- 
tions for the evaluation study? 

^. HOW ADMINISTFATOHS -AVE RESPOOTEO TC I>IIME KEY QUESTIONS 

How to go about the business of e^etting an evaluation study 
started is a different matter in different circumstances. Experi- 
enced administrators have different advice to give, partly because 
they work in different circumstances. Still, there is a substantial 
agreement among t,hem, ar- the responses on the following pages indi- 
cate. 

About l"^ administrators were asked to respond, in writing or 
orally, to the first five questions here. Some raised and answered 
other questions, and some referred to statements in the professional 
literature. Included is one statement from a'.i article in Educational 
Researche r by John Evans of the U.S. Office of Education, stated here 
as an answer to a key question, though !\r. Evans was not one of the 
administrators I sought out. 

None of the administrators has spoken here for his agency but 
only as an individual experienced with some aspects of initiating 
the formal evaluation of educational programmes. What was most ap- 
parent from the careful consideration and candid responses given to 
the questions was Zin optimism Irhat evaluation studies can facilitate 
the governance of educational programmes and that good lines of com- 
munication between commissioners and evaluators are essential. The 
answers were made with a quick readiiiess to share experience. The 
excerots on the following nages should help administrators boch with 
large and small experience with eva?.uation studies to focus on the 
responsibilities of negotiating productive v;ork:ir]g arrangements. 
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The questions started with this heading: 

Consider the advice you might pass along to educational offi- 
cials who are about to commission a large-scale evaluation 
•study. It could be, for example, advice to a West German 
government agency about to set up an evaluation of special 
finding of schools having new science programmes, or advice to 
certain Australian officials about evaluating consultancy ser- 
vices to outback districts. The evaluation is to be done by 
an outside group, not by staff from the same bureau or ministry. 

Making the Evaluation Plans Specific 

Question 1: Do you feel that the commissioners should insist on 

highly specific plans by the evaluators? Which is the 
greater danger, that the evaluaxors will focus too 
early on varj-ables and issues found later not to be im- 
portant, or that the evaluators will spend so long form- 
ing and correcting the design that the results will be 
incomplete and unsubstantial? 

Charles Beltz: Thosp commissioning a study should make sure 
that they are clear in their own minds as to what exactly it is that 
they wish to have evaluated. Often, the negotiations with the evalua 
tors are instrumental in clarifying the thinking of the commissioners 
This approach causes a good deal of frustration and argument to both 
parties, is very time-consuming, and makes unnecessary demands on 
the evaluators for repeated design modification. How specific any 
plans for evaluation must be varies from case to case, but if the 
commissioners have prepared a well thought out brief for the prospec- 
tive evaluators, the latter are in a nuch better position to develop 
an appropriate and acceptable design. Taking this as the basic re- 
quirement, there must be sufficient latitude for agreod modifica- 
tions during the course of the pro,ject. Rigid adherence to all of 
the original design, without amendments arising out of changing cir- 
cumstances over the time of the project, can make the final ouxcoraes 
£t best incomplete, sometimes irrelevant. ^ 

Alphonse Biiccinot PI;^r.r. nhould be clear, but not necessarily 
highly specific. By far the greater danfrer with complex projects 
is focusing too early on variables and issues which will be found 
later not to be important. 

John Bankn: fio. Thr- plans for the evaluation should be con- 
sistent with the way tne experiment has been formulated. The plans 
should leave room for f^rowlng. The danger of too early a focu3 is 
more likely, but both '^re likely. 
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Stig Obel: It depends on the sort of evaluation the evaluator 
is up to. A psychometric evaluation should be based on rather speci- 
fic plans or at laast on a model of evaluation with examples from 
earlier work of the evaluator, showing rather specific variable anal- 
ysis. In most casss the second danger, that of spending too much 
time on planning the design, is greater than the first danger. The 
commissioner should be warned of this danger. 

Bryan Dockrell; The keyword is flexibility. That doesn't mean 
ad hocery or muddling through. It means thinking about the possi- 
bilities carefully m advance, specifying possible techniques but 
keeping the options open for as long as possible. It means too being 
willing to add additional dimensions to studies as they become rele- 
vant. It also means having the courar:e to acknowledge that data 
gathered with great care and at great effort is uninf ormative . 

Sal\y Pancrazio: Partially. Some of the atipects of the plan 
can be hi.-^hly yppcific (time-line, professional staff to be used, 
available facilities, needs for subcontractint^, costs, etc.). Where 
options In methodology sxist, or where act:ions depend on certain re- 
sults or findin^TG, the need for .-^uch decir,ion points can be speci- 
fied, as well as the specification of possible options. As an eval- 
uator of proposals and as a reader of final reports, I have no^ 
"seen" the latter danger occurring. In fact, the time constraints 
related to fiscal year approprlr- tion? and exT>endi cures prevent the 
evaluator from spendin,^ too much time "formin.:: and correcting the 
design" . 

Ariph Lew\-: ".iiestion 1 should be divided into two portions: 
first about the specific plan ar:d then about the definition of varia- 
bles. The preparntion — n specific plan does not necessarily re- 
quire the early H..:-initi :i of th(^ variables. It may be that the 
plan contains a ti::M^ schedule vhich ?»llcw3 for exploratory studi-s, 
but which also reqinrer. th^t v riciMes will be defined by a certain 
date. Moreover, ir: thri t tlm^' evaluatnrs should provide "justi- 

fication" for the •jr.lect ion ai. • -^MUnition of the variables. .Such 
justification should be bnsed (^ii the findings of the exploratory 
investigatJ on. 

Generally the pl-^n should be hip-hly specific in a '•;ense that it 
should contain details about the data areas which should be covered 
by the stntiy. Examples of such areas are: cognitive behaviours of a 
higher mental t:/pe, transfer of knowledge, interest, co-operation 
among students, management problems, teachers' and parents' identi- 
fication with the program. The plan should indicate how thene 
variables should be defined. If they should be based on the rer.ults 
of an exploratory study, hov should this study be c. -inducted, how many 
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observations, etc., should be made? The plan should also be speci- 
fic concerning the sample size, methods of sample selection, and 
time-table. Such a procedure may avoid both shortcomings mentioned 
in the question, the focus on variables which are not important and 
on the attainment of unsubstantial results. 

Neg^otiating a Contract 

Question 2: Should the officials and the prospective evaluators 
meet and negotiate a contract that specifies the de- 
5i,?n to be used? Is it better to have a long period 
of negotiations? Does it help such negotiations to re- 
fer to previous studies, instruments, reports, and 
findings? 

Bryan Dockrell: Association between the various groups in- 
volved in a stKdy should begin early and should be seen as a process 
of mutual enlightenment. The evaluators will certainly have a job to 
do in clarifying t)r^ limits of the information that they can pro- 
vide. It is understandably difficult for administrators to realise 
that a mean score on a test may not convey the information it ap- 
pears to. The classic example of misunderstanding of this kind is 
the Stuart p.nd Wells readinr survey. rhe repor-*-. itself was impec- 
cable. It was nonetheless misunderstood. In these early discussions, 
reference to previous studies, possible findings, and appropriate 
instruments would surely play a major part. I do not see these dis- 
cussions as merely preliminary to a study but an importan^t part of 
any evaluation. In a study on primary school achievement in one 
Scottish local authority area, we had a series of meetings with of- 
ficials followed by the establishment of a steering committee which 
included representatives of a school system and those in the edu- 
cational community, the university, colleges of education and our- 
selves who were primarily contributing expertise to a process of 
self-examination. 

Sally Fancrazio: Thp nerrotiation period should not be long. 
However, I can see that v/here the proposals do not fit the spec if i- 
cations, and the best nronosal needs to be discussed with the sub- 
mitters, i n ordr-r tc: brin:^ it "up" to the sppcif icat:; ons , that the 
period of nerotintlon v/ould require a lonrer period of time. Most 
of us want to Iceep within ti^^ht time-lines. 

Alnhonsf? r^iccino: Officials and evaluators need to extend 
negotiatiopG over nr. uor.r a period an necessary to reach a proper 
un d e r s t a n a m p" . 



Arieh Lewy: The inquiry desi??:n should definitely be negotiated. 
Nevertheless the evaluators should insist on adhering to those scien- 
tific standards which they feel are relevant to a particular situa- 
tion. Thus, for example, if instruments should be developed they 
should not agree to use these instruments without proper validation 
procedures, even if this may shorten the time needed for the study. 
In such negotiation, examples of previous studies may be very use- 
ful. They help to clnrify flaws and merits of other studies and 
thus they may improve the utility of. nny new evaluation study. 

Constraints on Acess and Publicaxion 

Question 3: How important is it for commissj oners and evaluators 

to discuss and make specific agreements on such matters 
as to who will releas j information, confidentiality of 
sources, access to special places? Can you describe 
nny real i.n-stance in which failure to discuss such 
thin^^s caused a real problem, o*' where such an agreement 
was too constra j ninr? 
Jolin Banks: The government feels it owns any information from 
projects it funds. it expects to control it, even if it anticipates 
making the information public. It should be the government »s re- 
sponsibility to define cases in which it would feel Justified to re- 
strict publication. Two instances of such a problem come to mind, 
one an economics study and one a study of police records. 

Paul Dressel: The evaluator is trying to find out about the 
worth or value of some enterprise, but he also comes to this task 
with a set of values. The persons for whom he works and the people 
whom he evaluates also have a set of values. And very frequently it 
is the lack of recognition of these values or the clash between the 
two sets of values or priorities in valu^is which is the source of 
the difficulty, '^ile I do not think an evaluator needs to insist 
on a situation in which he can write up and publish everything he 
finds, or give interviews to the newspapers to insure that all of the 
difficulties and weaknesses that he locates are broadcast to the 
public, I think he does need to know whether the people with whom 
he will be ao.soK.iated will tolerate any critical comment. 

Alphons- Buccino: opecific agreements about release of infor- 
mation, and so forth, are absolutely essential. The real instance 
occurring most often is that certain information may not be made 
/available;' to evaluators ir: the absence of such agreements. 

Stig Obel: I'he nrobleni c' -e-er^sing inform^cion is important. 
The other two =i::rectR are not Mnnortnnt. rrequently a Dublished 
report revealin.T nnlv part of the picture - from an evaluation of 
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only part of the programme - has been presented as the background 
for further innovation. In almost every school you can find that 
reading and writii.g test results have been used by parents, head 
teachers, or local authorities to change the programme in the class- 
room. These tests are not reliable bases for changing the programme 
in the classroom, and should be used as but one part of a more com- 
prehensive evaluation. An agreement to withhold or release infor- 
mation is not common on this level, but here the dangers are clearly 
illustrated. In large-scale evaluation studies the problems and 
dangers are even greater, 

Charle.3 Beltz: The issue of publication rights is likely to be 
a persistent problem because of the . di f f erent motivations and in- 
terests of the parties involved, in the educational area, as dis- 
tinct from security areas such as defence, it would seem quite 
possible tc come to explicit agreements by which, the commissioners 
undertaKe to release any progress information and to publish the 
final report, AS SUBMITTED, and by which the conditions of access to 
the project's data, subsequent to completion, are clearly determined. 
Confidentiality of information sources should as a rule be maintained. 
In fact, it seems most appropriate at completion of the project to 
either destroy the information permitting identification of sources 
or deposit it for saf e-keepinfj with the commissioners. The guiding 
principle should be that the information is supplied for the speci- 
fic purpose of the project only and not to be used for any other 
purpose without explicit prior approval of the source, 

Arieh Lewy: if no generally accepted standards exist concerning 
publication procedures, then an agreement should be worked out. On 
the basis of frictions that have developed in the past over publi- 
cation of research in Israel, a schedule has been worked out for 
studies supported by the Ministry of Education. This scheme requires 
that first the results should be submitted to the commissioners of 
the study. The commissioners are entitled to criticir.e the study 
and to ask further clarification or explanation concerning critical 
issues. If, follov/ins this exchanf^e, the results are stated in a 
way that satisfies both researchers and commissioners, both sides 
are free to use the study in any form. If such an agreement is not 
reached the first pu^ :.c:ation of the study should include statements 
in which both sides .. licate their viev/s. This practice avoids prob- 
lems related to publication, /oiy deviation from this general prac- 
tice requires previous ap;reement. Generally it may be e fTood habit 
to prespecify ':ho conditions oi publication as they are understood 
by both parties. 
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utility of Research 

Question h: How common is it for the evaluators to set up a re- 
search study that will be admired by other researchers 
but not of -much use to the c:omniissioning agency? 
Ralph Lundgren: Quite common. The agency should obll/;ate the 
evaluators to be fully conscious of the purposes of the study. 
Evaluators neod to expl.Tin the implications of various research ap- 
proaches. 

Charles Beltz: This can cause serious problems where the com- 
missioners are themselves vaf::ue in what they want a project to 
achieve and, by implication, must leave most of the decisions on de- 
sign, Kind of information needed, and outcomes to the evaluators. 
The central factor cf a commissioned study must be its purpose; if 
it happens also to attract the acclaim of other researchers, that 
is a happv side-effect, but it should never be its purpose. 

Paul Dressel: I have run int--^ a good many evaluators who seem 
to think that the world was f^oinf- to be revolutionized as a result 
of their studies. The evaluator tends to feel that his activities 
are far xore important thar.- they may really be in the life of a 
larre-scale nroject, and again he may place very great emphasis on 
certain values which are of nuch lesser consequence to people who 
are involved, hor evaluation supported out of political delibera- 
tions, it is very easy to see why proponents of a particular project 
are dismayed by anything; in the way of a critical evaluation. Polit- 
ical decisions being what they are, any critical comment can be 
seired upon, blown un , and become the basis for throwing out support 
of a project - in order to ^ret funds to support something else. 
One of the rre^test problems is matching up the maturity and capa- 
bility of an evnluator to the persons with whom he works. I have 
been particularly conscious in this institution over the last ten 
or fifteen yearr that, ns an institutiomTl researcher and evaluator, 
I have arrived at the position where 1 can speak bluntly about the 
kinds of results loiLnd , delineate what^oui-ht to be done, and not 
feel that U\ the process I was endangering my owii stn cu'.5 or that of 
the orfice. In contrast, 1 have a Report on my desk, which just 
came in this morning, from a rese/rcher who admits that, because of 
his concern about critical reactions to certain aspects of this 
study, he consciously de-emphasi^ed these both in the process of re- 
nearch and agnin in the process of writing. The net result, I think; 
Is a report which evaded the anjor issues and which will make very 
little impact. If I could give any one word of advice to an evaluator 
as to how to look at the success of his own project, it would be that 
to find out whether he c^.used people to think more clearly and deeply 
rather than whether thev chanred or replaced the activity. 
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Arieh Lewy: This seems to be one of the most serious problems 
in the field of evaluation. The problem becomes even more serious 
because of the low standards of practice among evaluators. There 
seems to be a tremendous gap between the competency of top evalua- 
tion experts and the general practitioners. The practitioners try 
to adhere to prefabricated models and designs of study and frequently 
do net understand the nature of the problem they deal with. Such 
studies are often admired by their colleagues for adherence to models 
and for the utilization of complex statistical orocedures. The real 
problem is that in the field of evaluation there are no valid cri- 
teria for Judging the merits of an evaluation study, and frequently 
studies are admired without meeting acceptable standards of scien- 
tific work. Many evaluation studies published in journals lacK even 
.."internal validity". Journal manuscript reviewers judge studies on 
the finesse of the methodology without examining whether the methodol- 
ogy fits the problem of study. 

The Prestige Panel 

Question bi ^rfhere can an administrator find guidelines for using ~ as 
several of you and your fellow university presidents 
have - the "blue ribbon" panel to investigate a problem 
or to evaluate a programme? 

David Henry: .1 can confirm that (James) Conant, (Clark* Kerr 
and I have "advocated and utilized" blue-ribbon panels, task forces, 
and study commissions in the evaluation of complex programmes. How- 
ever, I am at a loss to refer you to any published conmient by any of 
us that would be useful in analyzing the process. I have personally 
served on four such national groups and feel keenly that they' are 
important, indeed, that they are an essential part of our governance 
structure in the formulation of public policy. T have not had 
occasion, however, to put my reactions on paper. Of course, there 
are many evaluations of individual reports, you know, and probably 
somebody has made a gnncial evaluation, but I have not encountered 
it. • . 

In my seminars on Current Issues and Problems, where I used the 
reports o.V n?^tional ,-roups as background material, T have introduced 
the subject about which you write, by noting: \} the special com- 
mission is able to focus on a complex ar^a and bring to its analysis 
the best in scholarr-hip and current informed opinion; 2) by its 
nature and the callbro of its people, a commission receives more than 
ordinary public attention and hence becomes an instrument in public 
information about nroblcmr nnri ir.sues. Sinoe public policy is or 
should be the result of extensive public debate, the commission re- 
ports elevate the converr^ati on beyond mere opinion gathering and 
place it in the f ra^icvor --. of objective -^.naly.'^is nnd ovaJuation. 



Although some refer slightingly to reports of task forces that 
"gather dust unnoticed find unread", oven a casual study of what 
really happens to recommendations is quite revealing. For example, 
all of the recoramendations of the Eisenhower Conmlttee on Education 
Beyond the High School found their way into acceptance within the 
decade following the report. This is nor to nay that the ccmmittee 
report created the ref.ul't. It aoer su^^rest, however, that the com- 
mittee may have helrec formul-Hte tne nature of the result and its 
timin^-l. i;o one, oi course, could ever be sure of cauF.e and effect 
in such an instance, iiowever, the C?.rnei:ie Commisr.ion on Educational 
Television resulted directly in the adoption of the r\iblic reievision 
Act of That tracing is clearly in the record. 

Another dimension of the task force is at the state level, by 
institutions and state {^nvernments , and at the local level by munici- 
pal r;overnance. It is interest! njr that some states seem to mal'.e 
rrroater ant! rnore effecrive \iso of the blue-ribbon panel approach 
than other's. Aaw ''*or.; is a sti.i:5y in contrast with Illinois, for 
example. The 'lew Yor\; D'lttern uses "the blue-ribbon" panel; the 
Illinois pr act Ire Tnvors the reTM*e tentative panel, without so many 
blue ribbons ntt:iched. Tho rorrr.er, of course, is always more pro- 
ductive because there arc no nn'^cial intf^rests involved. The. latter 
tends tc renort the lowest c;M:un.-in denominator of consensus amonf; con- 
flictfu.V rrouns. ^cr that reason, politicians 11 ^:e it. 

For Eva lu r? u i n ■ " I nn ^ vr> t \ on s 

';uostioi: v:ri.T t r>re the e?;spntin! t>riturei: of an evaluation of an 

i n p. c) '/ a t i v e p r o ^ r-' 1 1 n m e ? 
John jisbet: : think we can safelv di^-card as inappropriate 
the or / • .1 nn I Xz.er i r a n *n.od c 1 of p re c i s e v d e f i n ed beha v i oura 1 ob.j e c - 
tives, meas'jrable outcomes, st:^.ndardi?ed tests and elaborate statis- 
tical a na ' v s i s . i h 1 s v.'n r '.mma t i ^t- e x te ma 1 e va Lua t i on : sufTanat i ve 
in mat it wns .?pr>!'. ^^ "-'r..- en'i ( '.'^r r--' :nt- * i rnes only at t*ne end of 
the initial st-^ 'o : oxtcrnal in that It was done by a rroup separate 
from the Innova to rr>, supposed Iv neil|ral but often unsympathetic. 
Instead, v/e need a continuoi:s or on'';oinjT. ev'iluation, built into the 
innovatorv nro-'rarr.nin ^s an inte^^rnl part of the team work. This is 
the model v;hlch w^c developed in the orruv/LS f'louncll Project, 
r.nd the iiurrianities ''urr ical-um J'ro.i'^c t . 'i he eva i.ua tor is an Important 
riienber of the u-'^in, and he hns one of tne most difficult .jobs to do. 
He is invoIv{?d thr ,»i.!''hnj t , in the planning (to ensure that evaluation 
is possible , du!'inr -he pro 'rp'Mme ''to ensure that relevant records 
are kept. , and at the end jf each stare ( "C -rc/ido fr-cdh^C: to the 
team ■"<n their stren.-vlhs ;ir,c v.'oa :nesses . It is a pattern of forma- 
tive or responsive eva lu«''^^:i nr.. 



Perhaps this may seem to allow too much freedom to the innova- 
tors. How do we deal with those who say, "It is clear that there 
has been a change and we believe that it ii:-, a ch^nf^e for the better", 
or, "We know from our day to day contact that the children are learn- 
ing more effectively, and our statement of that conviction is the 
only valid evidence''? This is certainly part of the evidence, but 
the i/eakness of this style is that It denies to others the right to 
use their Judgment - it withholds evidence. Evaluation is not only 
a Judgment: it also izetc^ out the evidence and reasoning which led 
to that judgment; and if evaluation is to be accepted as valid, we 
need to be sure that the evidence reported is a fair sample, and 
that the reasoning from it is logical, and that alternative inter- 
pretations have been considered and disproved. There is no neces- 
sity for statistics: recently published evaluations have included 
case-studies, transcripts, specimens of pupils' work and so on. 
Evaluation is a form of communication, a sharinjr; of the experience 
with others. Mowadays, exneditions into high mountains or jungle 
include a cameraman in the team; the choice of camerar:ian may be an 
important factor in winning; public synpathy with the aims of the ex- 
pedition. 

puoted from "Innovation - Bandwaron or Hoarse?" a naper pre- 
sented at the I- rank Tate r-:emcrial lecture. .'-eibourne, Australia: 
Mo n? s h Un i v e r s i ty , July ? , 1^7 ••■» , 

Genera L Ir. Inr from Studies Elsewhere 

Question 7: Xcst of the examj.'] es of evaluation studies of large- 
scale educational prot::rammes ccme from the United States. 
Some European officials doubt that studying those cases 
would help them negotiate a contract for evaluarion. 
'A'hy? 

Bichar-i Hooper: T suppose that they could be saying that if 
you look nt America, nnd the educational culture there, there are 
some raa ^or differences: the whole notion of contracting v;ith any- 
body to do anything: in education, the notion of project propo:,-.'Is, 
and feder.il fundin^:;, p.nc] the rcrd I-':":::dat.i on , and nro '"essi onal nro- 
p r ■ s r Iter s , an! , , , That Is r. t i !. 1 f o r e^ .j n , f o r e i / -jn moon land- 
scape, for us in Ku^rland . 

::ow you see, not only is it forpi,:n here, it ir actively dis- 
liked. It is seen as not geri t:le:r;rn"! y » not quite on. rather corr.mcn, 
a bit ^nrlgar. Hducatiop. isn't li/e r::pz, I thini^ in Ki:ror»ean coun- 
tries a lot of 'leci:', ion- nro macJe "nt tne club," amongst gentlemen. 
The practice - if you like, the democracy - of prooosal writing, it 
doesn' t go on hor*^'. Ther^ is n'ji*..? a democracy in i *■ , I'ncv;; 
a s ^ o &y can slap in n r'o p c- - . r» !. t • i t S e f e d e r a 1 -* r v i- r. r. . People here 
ar'n't '''.nov.' that they co\l'': : :.o;iev fr'om Tj^Z ( thh Oenartment of 
'.f.d 13 c T : on r^'K* 3 c i e n e . 
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So I think that is a major difference. And when you talk about 
contracting with evaluators, you are immediately assuming that that 
as a concept is familiar. It isn»t. Let»s realize that there is 
not much experience with evaluation studies here. 

I think, secondly, that evaluation as a sepa rate activity is 
totally foreign to most people working in the established parts of 
education. Someone comes along with a thing called "Evaluation"; 
and when he starts describing it, they say: "Don't teach your grand- 
mother to suck eggs. We already do that. You are dressing up what 
we would call common sense with fancy theoretical terms." It is 
like the response to the sociologist in this country. A sociologist 
starts talking, and everybody says, "All a sociologist does is point 
to the blindingly obvious." And evaluation is in the same dangerous 
area. 

Evaluation, as a thing to talk about, is foreign. It is there- 
fore threatening. Especially this evaluation talking about how 
people make decisions. Our countries are not as open es the U.S. 
in many ways. The U.S. is much more open, in a sort of investigative 
Journalism, Watergate type of world. That does not go on in Europe. 

You want thera to study an issue that - they hope - isn»t going . 

to be an issue. 
..« " 

Obs ta cles to Federal Evaluation Studies 

Question 6: In recent years what new problems have arisen in the 
evaluation of federally supported educational pro- 
fi;rac:raos? 

John W. Evans: 1. As educational research and evaluation 
h?.ve proliferated, the oeopie and institutions who are the objects 
of these studies have come 'jnder an increasing data collection 
burden - and are increasingly expressing their rer^istance to it ... 

2. Evaluation studies that involve collecting data on adults 
are erxounterint: increafiinr lonistance the interviewee level, 
particularly among minorities and the poor where it is now not 
uncommon for respondents tD insist that they be oaid for their time. 

3. The increased sensitivity to evaluation studies - both 
what they seek to find out and the amount of data they propose to 
collect - is resulting In a ^tran^lin,^ growth of reviews, clearances, 
and advisory bodies .... they threaten to prevent many evalua- 
tions from beinp; carried out at all. 

U, As protests over evaluation arise, ostensibly over the type 
rt-.'J aii/int of data to be colle^-^^fvi , there 15; likely to be an in- 
crt;.:i-ed politicization of these protests hnd their use r.s weapons 
in broader disputes betveen local and federal levels of government. 
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5. As evaluation activity and policymakers » interest in it 
have grown, there has also been an increased awareness at the pro- 
gram level that it is necessary to start taking evaluations serious- 
ly. This has had the unfortunate effect on some program officers 
and school administrators of increasing their unwillingness to par- 
ticipate in evaluation studies for fear of what will happen to their 
programs if the evaluation produces negative findings. 

6. Evaluations are increasingly encountering unrealistic ex- 
pectations on the part of policymakers and legislators with respect 
':o both the speed with which evaluations should be mounted and com- 
pleted, and the simplicity of the answers which are desired. A de- 
mand for evaluation has been created, and it is an increasingly 
insistent one. Policymakers are beginning to display an irritated 
impatience with the elaborate trappings of careful design, longitu- 
dinal .-studies, and complex multivariate findings. They want to 
know whether or not a program is any good and they want to know it 
yesterday. As unrealistic as these expectations are, evaluators 
themselves probably must bear some of the blame for them. In their 
early zeal to have the virtues of evaluation recognized and used, 
evaluators were almost certainly guilty of overpromising. 

7. We are certain to see a lot more public debate of the kind 
I noted earlier over the validity of evaluation methods and results. 
An increasingly important and t.Tme consuming task for evaluators 
will be defending the evaluations they carry out and their suitabili- 
ty as a basis for policy decisions. 

Quoted from "Svnlur^ting Educational Pro-rams - Are V/e Gettine; 
^jnvher.:-", Educational Researcher, Voli 3, Mo. 3 (September 

Advice to CommlssioniniT Athene ir .^ 

Question 9: Wh-^t ndvice would you sav in most important for com- 
missioning'- officials? 

AlphonsG Huccino: (a) ::iP.rify the purpose to be served by the 
evaluation and the ran.-e of fictions to bo offectec. Get involvement 
of individuals likely to "us-" the evaluation results. (b) Be sure 
that the profrrnm or pro jeer to oe evaluated is defined wif- a Cur- 
gree of clarity conmensum tn v. ith the degree of nrecision r O-'Cted 
of the evaluation. If pro,-r..m outcome-:: nvc :rot well defines, -.^re- 
':.i.re measurers should not be attempted. (c) Cost of evaluation 
snould be closely related to the ootential value of evaluation re- 
i-^'Jlts. A lov-cosl- sur.ey -rn, in pome instances, yield Mif or.-nation 
as valuable as th?.t C(r.:iinr from a larger effort. (d) At,oraise the 
evaluator'3 capnbilitv for interpi-etntion of dat.n. Data in ard of 
themseJves are not very useful in the absence of insightful inter- 
rrota t ion , 
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Bryan Dockrell: Make sure that you understand what the evalua- 
tors can provide for you and what they cannot* There is still a 
touching naivety about the faith of some administrators in evalua- 
tion, a naivety which if not fostered is certainly not discouraged 
by some researchers. 

Charles Beltz: (a) Do your homework in preparing a brief for 
the evaluators. Take the tiice and make the effort necessary to de- 
termine: 

what exactly is to be evaluated; 

what is the ourpose of it: increased knowledge or action; 
if action is the purpose, what stenz are necessary to pre- 
pare for effective implementation following completion of 
the pro.ject; 

what financial resources are likely to be available; 
when are the 1 indinps needed, i.e., what are the time con- 
straints ; 

.. what is envisaged in respect of publication, conyrights, 
confidentiality , etc . 
(fc) rind a mechanism for a continuous relationship with the evalu?- 
tors during th*- wholo rieriod of thr^ project. This is a vital aspect 
of a commissioned pro.lect of any size or duration, and I personally 
favour a steerine /^roup or committee convening regularly to: 

.. maintain contact with the project; 

.. receive and discuss progress reports; 

.. act as a forum where either party can air problems, conflicts 
or new renuirements for discussion and settlement; 
••jdvise the evalviators , ?A their own requer,*: or otherwise, on 
any aspects of the project or relevant new developments or 
poli::y decisions. 

/he comoosition of such a grouo would be a point for negotiation but 
•naght include?, in addition to the two parties, snme other persons 
representing, for example, interest groups or professional expertise. 

Stir Obol: Find the ri/rht blend of evaluation (psychometric, 
sociometr^ic, rToal-free, and oth-r component:.-) to give a new pers-;ec- 
tive to the oroblom. 

John Banks: There is a problem selling new evaluation ideas in 
government. Officials leel competent to deal only with certiiin 
ideas, issues, and innui ry methods. There is a tendency to leave 
the rest to others. But if questions of one kind only go to offi- 
cials of one kind, Ow.-., pr.onomic nuestions to economist*;:, then we 
have a terrible mess on our hands. V-e need ways of encouraging and 
protectinr arencies for revi swing issues and programmes that extend 
outside their 0'fn\ sp.^clrilizationr. 
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Frederic Moeher: Consider the particular circumstances. There 
aren't any general rules yet. 

Start far enough ahead to allow yourself to commission a design 
(or competing designs) which could be worked out by the evaluators 
in conjunction with the evaluated. 

In touchy situations officials and evaluators niay want to con- 
sider adding a weighty advisory group to endorse or second guess the 
process and the results. I suspect there is a trade-off between the 
forcefulness and originality of the work and the safety of numbers. 
You have to be sensible about the balance. 

Plan ahead. Take your time nnd think out what you want to 
know and why. Get somebody sensible to help you think about those 
questions before you plunge in. If the odds are against getting 
useful answers, don't do it. 

Arieh Lewy: Play a simulation game, see what alternative re- 
sults might be obtained by the evaluative study, and explore the 
actions to be taken for each different result that might obtain. 
The evaluation study should be structured in a v/ay that the practi- 
cal implications of different results will be clear in advance. 
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The admipistratorn who answered questionnaires or provided state- 
ments to permit the forogoing summary were: 



John Banks 

Department of Education and 

Science 
London 



Richard Hooper, Director 
National Development Programme in 

Computer Assisted Learning 
London 



Charles Jeltz 
/-ssistan": Secretary 
Department of Education 
Australia 



Arieh Lewy 

Ministry of Education 

and Culture 
Israel 



Alphonse Puccino 
Groap Director 

Problem Assessment and Sneclal 

Studies 
National Science Foundation 



W, Bryan Dockrell 
Director 

Scottish Council for Research 
in Education 



Paul L. Dre!!sel 
Assistant Pr^.ivoi^t for 

Institutional Research 
Michigan State University 



John W. Kvans 

Office of Planning, Eud^^eting 

and Evaluation 
U.S. Office of r^ducation 



David D, Henry 
President Emeritus 
Univer^oity c:' Illinois 



Ralph Lundgren 

Eli Lilly Foundation 

Indi.anaDolis, 

Indiana 



Frederick A. Mosher 
Program Officer 
Carnegie Corporation 
of New York 



Jolin Nir=bet, Head of Department 

of Education 
UniverL.ity of Aberdeen 
Scotland 



Stig Obel 

Inspector of Education 
K.mistry of Education 
Cs';nenhc^ -. en 



Sally F. Pamcrazio, Director 
Department of Research and 

Statistics 
Tllin:):.': Department of Ediioation 
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U, HOW RESEARChKr; MAVE ".E3P0NDED TO SEVEN KEY QUESTIONS 



There ?re differ encas amoutr, researchers, as there are in any 
group, as to the best .uethods i'-.-.r- getting the job done. This per- 
tains, as well, to the ,iob commissioning an evaluation study. 
Some researchers prefer r^rv formal ne^otiaticns , with a careiul 
specification of i.he purpose, the procedures, and the responsibili- 
ties. Some prefer a minimum of specific procedural commitments, but 
well-defineu methods for monitoring developments and resolving dis- 
agreements. And some prefer the least ackjiowledgment that there a: 
expectations an:i obl?-gations on both sides. 

Many thl.igs happen in the couroe of a programme evaluation study 
thnt change the nature of the nrogramme and its evaluation. Some 
anticipation of these cht^nges is necessary. Some discussion of them 
between commissi one:* and ovaluator is recommended by almost all ex- 
perienced pv.rties. But th?re is disagreement as to how many poten- 
tial calamities should be considered, and more disagreement as to 
how formally they should be recognised in any contract or agreement 
made. Both parties nead protection when things go awry. It is 
best to get that protection spelled out early. But most problems 
cannot be adequately anticipated, and the very discussion of potential 
problems is likely to get the focus of the ©valuation shifted, and 
the range of issues narrowed, so that the evaluation may be dealing 
with them, sometimes properly so, and sometimes overly so. The 
overworking of these possibilities may cause the purposes of the 
evaluation to become limited to the superficial things less likely 
to become troublesome, and the issues may not be those that really 
concern th3 audiences. 

In order to identify now researchers respond to this sort of 
situation, a small tA^oup of them ware asked se/en questions. From 
the lengthy and thoughtful answers they sent in the statements on 
tha following were selected. 

Choosing the Methods of Inquiry 

Question 1: The rese?\rcher feels that sometimes a sponsor prefers 
a form of investiriation that is not suitable for the 
issues at hand. Should the evaluator be r;uaranteed the 
freedom to choose the methods of inquiry to be used? 
Is this an imnortant issue? 

David Hamiltoni Yes - an important issue. I don't think the 
evaluator should be ruaranteecl the freedom any more than the sDonsor 
should be ^-uaranteed the ri^^iht to determine the research (V/ho are 
the sponsors anyway? The Ford Fojndation? The shareholders? The 
workers? Or those vho buy the carz'^ ) , I think that the sponsors 
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have the right to spe ,ify the problems they woii:'.d like addr^ssc^d, 
then it^s up to them and the evaluator to decided upon the means. 

Dan Stuffl9^eam: This obviour>Iy is an important issue but my 
answer is no. i think that any .evalua-cion desipTi is a colle;.:tion of 
decisions concerning many thinfrs including, audiences, Questions to 
be addressed, authorship of reports, editoric^i. license, delivery 
schedule of evaluation oroducts, schedule of oaymenLs, accens to 
personnel ard program data, ri^ht o^* rebuttal, and so foi'th. These 
decisions can r;reatly affrct the vcltare of prof^rams and pei-sonnel 
as well as the qns^llxy and utility of the evaluation work performed. 
Hence I think that -these ite-.is should be considered ne^-^oti' ole and 
that representative's of the >T;rouos to be involved in and affect-^d by 
the evaluation should r.iF:otiate the items in advance- of conducting 
the study and that the resultant contract should serve as a guide 
for the studv. Of course, the contract should include a clause for 
allowing rev and renef;otiation . Further, any evaluator who can- 
not reach acceptable a,£:reements with clients can refu^je to enter 
into an evaluation relationihip or il' "Iready employed by the agency, 
can quit, :^oth r,uch e- o:it:% have occurred several times in the oast, 
but probably not oiten enouA^h. 

Thomas Covens: The choico of methods of inauiry is an issue 
that should be settled by negotiation between the evaluation con- 
tractor and the sponsoring agency. In some cases invo;vin'/ competi- 
tive bid, the evaluator 's proposed choice oi methodology may be a 
critical factor in deciding which evaluation -agency is selected. 
The agency prenaring a RFP (Reouest for Proposal) may also have 
strong nreferences for a partic.:lar methodology and if so, shculd 
specify them in the RFP. Obviously some methodoloc::ies are less aporo- 
Driate than others for ansv/ering certain key ouestions. A sophisti- 
cated government bureau should probably kn :w in advance the Key 
questions it v;ants the evaluation to answer and which methodologies 
would or would not be adecuate to answer these Questions. Also the 
funding agency may need t? express any constraints that v/ould rule 
out certain methnr!s of inquiry for te'^hnical, political or financial 
reasons . 

Marvin Al"-:in: I think the evaluator has to have the final say 
regarding rr.ethodologies . -he agency ha s to find its pro tec t ion in 
on-tiir.e submission of a technical comr t^tc and correct reoort direc- 
ted towards decision concerns prespec*. Tied . 

Alan Thomas: Thi- evaluator should h.^vo considerable Ic-titude 
in selectin/t the method to be U30d. In extreme? coses ho should re- 
fuse to undertake a task if ho is constrained to use methods he 
feels are inciporoorla to . 
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Karl Krey and Manfred Lanf^: Some points have tc be discussed 
and decided In co-operation with the sponr-cr and will partially in- 
fluence the choice of methodr.. 

In jTeneral the ev?/Iuator should :vwq the Treedorn to choose among 
possible alternatives. ae knows ^he rros and cons and validity of 
methods. 

" P i ^ -try-bacl: i n^;" 3asic :\esearcri 

Question 2: The evaiuator sometimes says that some of his research 
effort must ^:o into the basic questions of education, 
.'jhouid the evaiuator asV. for assurance that some of the 
effort in each study be devoted to basic I'esearch? Is 
this an imoortant issue? 

Krnest House: The evaiuator may ask for some resoiir ;:es for 
basic research but the sponsor is under no obligation to provide 
them - and should not do so most of the time. Becoming fixated on 
research issues is likely to lure the evaiuator from the particular 
focus oT his task - the particularities and peculiarities of the 
project itself. 

Fvobert Hippey: The issue is important. However, I think that 
it is more iu^r3ortant to i^et competent, inquiry-oriented evaluators 
furicti.onin,^ than it is to ImDOse specific basic research exDecta ti ons 
on a oarticular pro.)-?ct. In some instances, it may be more ImDor- 
tant to implement a promi.sinf: idea nnd study it once implemented 
than to nreordain basic research objectives on an entity which T-ay 
never fly. i'Jesearch can often appear threatening prospectively. 
Perhaos better research can be done once rapport is developed, and 
Che resec!rch'.-rr ha.:- r-- " r*- intimate understanainp; of what he is 
studvir.'^, ihan he- ' advance. I am currently exrjlorinr a fas- 

cinatin;T researc:. ■ " " '.'<:. ".isin;: some data I v/ould not have had ac- 
cess to if I had ' T-:iV'' :'f>search assurances in advance of my 
invo 1 Vo>r.G::t. '<!est:^': Assurances - dysfunctional nt times. 

r-1alcolm Parlett: It deoends on personal preference. Personally, 
I think that the proper study of education proceeds through studies 
o.f ma : 1 f e oduca t iona;: -phenomena and proh'..ems encountered by edu- 
catorj. To do ^:ood aru^l iod work inevitably moans raising; basic 
Questior./, o: edMcatioi:. 'n oLhc-^' words. It is a false dichotomy. 

. A:rT':^\' : ov i : it Is and is not an important i ssue . T t i s im- 
nortanl :'ro-:i tno ri(>r oti ve o' the university based researcher, who 
needs to have re sea rch nay o if in a career* sense . it i c also i moor- 
cant in that availabl-' funir. oc'h:. to be used in the most efficient 
mann-r, and if a b.-^r-ic nurntion can be nosed while contributinr; to 
the vat ion , so "luch the t;ef:er. '.he i^ey is the word basic. If 
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basic means usin;^ the evaluation situatiori to test some academic 
theory, where, the primary Interest is in the variables ol" the theory, 
then I would say the inclusion of r^uch matters is secondary from the 
point of view of the sponsor of the research, iiowever, if some 
really .-riticnl question can be readily incorporated into the re- 
sear.-.h design, and have payoff for the SDonsor, xhen it should be 
done. For example, Peter Bryant makes a simple .r.r.estion in his 
new book that the teachin?-^ of readinr to yourvr children mii^ht be 
facilitated by the simple expedient of puttinr a red line down the 
left side of the r'age to orovide n frame of reference for left 
right spatial orientation in yoimR- children. The left right orien- 
tation is poori-y developed. That sort of idea could be tested 
readily, it has nracticnl innl ications , and it could be incorporated 
into an evaluation desi.^n with very little cost. That kind of thing 
should be done wherever feasible. 

KnlDh rvlr?r: An evaluation organisation should have a research 
proirram to which each evaluation effort should :nf.r:e some contribution. 

A]. an Tho-n^s: I believe the researcher has the ri^ht and tne 
responsibility to ensure that some of'his effort is devoted to the 
advancement of knowled^'ie. /\n important issue. 

Publishin.^ in Journals 

Question 3: The evaluator sometimes wants to publish results in a 

orofessional .journal. Should there be any restrictions 
on what he mif'ht oubxish there? 
Dovid Cohen: 1 can see none a priori , thouH;h agencies ought 
to have the right to reply in the same issue of the journal. 

Julian f.tanley: It seems to me that usually an evaluator should 
be free to publish in professional olaces articler or even books 
based on his evaluation, after a report of it is submitted and dis- 
cussed. 

Murray Levine: I think th?> only restrictions should be those 
consistent with confidentiality. T have on sevei-al occasions made 
an agreement with those involved that I would submit what I write 
for prior reviev with the understanding that I would control what 
was included mo what was Ef>ir., L did agr^e to include in the body 
of the work itself disagreements of interpretation, or statements of 
explanation so the oarty affected vrould have his or her interests 
represented. I have found that an effective tactic in relation to 
evaluation reports, and in relation to publication of a report. 

Ralph 'i?yler: Generally, knowled^^e grows from publications. 
The evaluation organization should clcirify in its contract what free- 
dom of publication is appropriate. 
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Mnlcolm Parlett: This is obviously somethinr; that can be dis- 
cussed in advoncc, thouz-^h the evaluator will often have to nut up 
with the sponsor sayin^-^, "let's wait and see, until we have had a 
look at what you are going to write". My o^-n feeling here is that 
the evHluator does not have to take the assignment if he/she does 
not like the conditions imposed. On the other hand, he/she can seek 
to persuade the sponsor to provide of^rmission for publication 
- especially on the .<;roiLnds that wider cornmunicati on of educational 
experiences is necessj^ry. 

Robert Hipney: I can work with some restrictions and not 
others. The conditions should be clearly stated. It is the evalua- 
tor's responsibility to draw up the specifications. 

Thomas C>wens: If the professional Journal article is based 
upon data that have not becoi:ie public domain I feel it is approriate 
for The evalucitor to [ret approval from thr contraccing agency for 
such Dublishin-t. .some cases t'ne contracting agency may wish to 

rele?se certain data prior to allowinr; therii to be published privately 
by the evaluritor. 

Krnest House; • There should be restrictions on Journal articles 
but these restrictions should be negotiated in advance. In particu- 
lar, the sponsor shoiilc hnve no rir-rht to censor Information in the 
renort unler.s the a '--reed-u-oon Pi'ocedures call for such editing. For 
the sponsor to be the final .-arbiter of what shall see the light of 
day is not healthy. 

i'ialcolrp. Pr-ovus: Yes. Client confidentiality - within the 
terms of his contract - co:nep first. 

D i s c 1 o s i n ?-.a I fe a s a nc e 

Question U: The evaluator sometimes finds something that he feels 
r.ust bo reported to higher authority or to the public, 
even though his contract calls for confidential treat- 
ment of information. Should ti.e evaluator ask for 
aF.r.urnnce that he can follov; hi::' conscience in such 
instances? 

V.vnne :iarlen: rollov;in*' one's conscience is probably indepen- 
dent of contracts, i.e., one would risk breaking a contract if an 
issue was sufficiently important. However, to avoid confrontation 
it v;o\r:d b^^ hcizz i* the ~-valu.?tcr v;ere formally given the right of 
decision as to whether information which others prefer to be kept 
c onf 1 d 0 n ti a 1 s houl d be r ep o r t ed . 

'Julian Stanley: This ic^ a tric-:y matter and can be answered 
only "It all deoer:':- ..." Pres^inablv, his evaluation should not 
be keo": secret or ho-vily edited, but in most instances it would not 
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seem desirable for the evaluator to f.o to news media or others to 
soring his findin/^s before they are reported to the sponsor. He 
should, however, have ?.SGur?inooP that the report will be made pub- 
lic within a reasonable len^tth of time after .it is submitted. It 
seems to me that secret evaluations by professionals are seldom 
justified. 

Malcolm Parlett: This possibility certainly has to be r^.ddressed 
in advance. Often it may be hypothetical, but possi bilit: ec such as 
these should certainly be paraded early In the rame. The evaluator, 
in my view, has to make It clear that he/she has to exercise his/her. 
very best judrnient in oupstions of ethical responsibility. There is 
? 'i^rev area' v/hL-i-c it is impossible to anticipate each and every 
moral conundrum, and thin has to be ackncwled^ed openly at the out- 
set. Obviously anv breakin.-: of the confidentiality 'contract^ vould 
have to be in extreme c ircumot^ncef. , and examples of v;hat these 
inir-ht be c;:-ii'.c: be raised by the evaluator in the preliminary dis^cus- 
sion. 

D-isn Sm'fLebeam: The .-'nswor to this one denend:^ on the prior 
contract that was ne/^otiated ynd on provisions for renegotiation a:3 
well as basic moral nuestionc that mi^-ht be involved once the find- 
ings are knovm. Generally I prefer the case where the contract 
gives authority to the evaluator to release his rencrt whether or 
no'^ the report is endorsed or appreciated by the sponsor. In this 
case, of course, there is no problem, at least not the kind you are 
.referring; zo in proceedin.p; to release the information to the public. 
In another case where the evaluator thinkr he oup;ht to release the 
information because of political pressure, because of possibility 
for pervasive impact, etc., I believe the evaluator should not be 
the sole decision maker if he has previously a^^reed to limit his re- 
port to certain defined audiences. Instead, I think he must reopen 
the neso-ciaticn of the contract and seek to reach agreements v/ith 
the other parties concerning whether the information should be re- 
leased and then should honor such agreements. Of course, there are 
times when questions of basic morality such as emerged in the 
vvaterr^ate mess present themselves. Then I think the evaluator must 
do what he believes, br>sed -on his own conscience, must be done for 
the welfare of society and in consideration of the interests of the 
per-sons to be affected by the release. 

Haiph Tvler: ICo one individual's conscience should be accepted 
as valid. The wav in v/hich information will be treated should be 
part of the contract. 
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Attention to Evaluation Findings 

Question " The ovaLufitor is soraeuime<i discouraf;ed by the little 
attention r'iven his findings. Should the evaluator 
ask for assurance that his words will be heeded? 

Julian Stanley: Realistically, would asking do any good? He 
can query the agency officials carefully before he contracts with 
them, but they may, nevertheless, try to bury or ignore his findings. 

Dan Stufflebeam: Thi,^ is a very tou^^h question, highlighting 
one of the most important problems evaluators have. By and large 
1 think our methodology is poor for communicativig with audiences, 
for working with audiences toward the application of information. 
We have too often felt that our work was done once we had prepared 
a 500-page report and sent it to the sponsoring agent. To provide 
some minor improvement, I recently proposed a new definition of 
evaluation that would, I think, place more attention on the communi- 
cation proces.s, The definition I proposed is: 

Evaluation is the proce.os of delineating, obtaining, and 
, apply inr descriotive and judgmental information for 

decision mnklng and accour.xability . 
Whereas the "obtr"?ining" steo is intended to be the technical ons of 
collectinf:, orf;&nir:ln.?: , .?nd analyzing" information, the "delineating" 
dnd "applying" steps are Interface ste^js. Of course clients, evalua- 
tors, and other interested parties mu>; c interface early in the study 
and periodically throughout to effect and evolve the airreenents that 
guide the study. As to the applying seep, the evaluator has the 
difficult job of providing information through writing and dis- 
seminating reports, through face-to-face meetings, through dis- 
semination of finding's through the public media, etc. At the same 
time the audiences an^i the client particularly must iiti'..lr:^- the in- 
fornation (or i;. re 11} :n regard to decision mak'n^, aoco'on'^sbili- 
ty , public relationi;, and so forth. My feeling :K'^t v/e ha";c' to 
do a rreat coal nore than v/e have done to incorporr ' cor^tiunication 
theory r\nd change theory into the methodology of cvr= lust ion. Evalua- 
tion, after all, •essentially is a change pi'oc;ess. If it is to pro- 
duce improvements or cr.anres of any kind, it seems reasonable that 
the well-r:n:}wn change nrinciple of involvemenx of those whose be- 
havior is to be affected ou&-ht to be incomorated into our plans 
for apnlyin-'^ evaluative information to the larger chanre process. I 
guess I would sny triat thi s problem ai'ea is probably the number one 
priority Item tha t evalun tors ought to be at tending to . 
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Robert Stake: The ovaluator may own his words but he does not 
own their me..ul:.r. The meanf. rr. the reader ^Mves - based often on 
more relevant experience - ^^ro more imnort.int than those the evalua- 
tor ^:ives. The evaluatcr should not presume he knovs best how much 
his words are worth. 

David iiarnilton: 1:' th- .-,-Iuator takes care tr -^c'dress the 
pertinent auestlous, ther. there is a much rreater chance that his 
findings will DC attended to. To my mind the choice oi questions to 
focus upon is the :nost critical part of the evaluator's Job. 

Wynne harlen: An evr.luator has no more rirht, purely as a 
right, to have h:s words heeded th.an anyone else. He must present 
his 'fif.din^TS so that their claim to be heeded is self-evident. 

Abuse of Evaluation 'rindin-rs 

rjuostion - : The evaluator soir.etimes imds that his findings are 
used as if thev sunoor:?d claims that they do not. 
:;hould the evaluator ask for assurance that this will 
"^ot happen? 

^'-^n Tr'-a'^-: ':e:~. However, it often happens that persons over 
whom he has ne control will misinterpret his results. 

Julian Stanlev: Kron his sponsors, yes - ard if they do so 
anywav he should attemrr to rebut their inaccurate cla-ims. I do not 
see how one ca:: prevent the nress from distorting its reports of 
evaluations, and it is dif/icult to sav to what extont the evaluator 
himself should t-v to correct mi simpressions created by the press. 
That mi-ht reouire mu-n of his ( tinreimbursed } time and eflo-t, with- 
out sufficient Dro Sessional oayoff. 

Ralnh Tvler: it is not legally possible to ask in advj.nce, but 
he carassure'^the :^no:-cr th-t misinternretations will be made^tablic 
i r thev f^c.ur. 

r-iaicoi:: Pirlett: This is difficult. If the evaluator be-ins 
to r-is- ':u-st:-n- li-^o this, >i-/she is unlikely to win trust and 
confidence* i do not tnink there is much that can be done over this 
p hen on en..n. 

K-ie'^- ''oust-: i'he evaluator must take care that his report is 
•'ot censored, e-ited, cnsiired in substance without his knowledge, 
u^;- i-iv,. o ' ^Pt, -r- o.-^nnct 'jv^ornire all interpretations and all 
uses^to'vnic:, his v/or- will no n':t. :ie may wish to refute some 

^ n-v^ p'- be wronr in his nvn interpreta- 

■» ntp^rore .a t: ons \,'.n-..^c .. n.i i.. 

t^on " ^-^^rrn^n^v thcu'ii, ne -ann^t wash his hands of all responsibili- 

t-!^/"-;^" niav .o-h tr. rua-d --Ir-t his w^rk beir;^ used in particularly 

. , , -« - M-o-' ir'^ure neonle, 

unsavory ^ ■ ■ 




Thomas Owens: The abuse referred to in this question does, of 
course, occur from time to time but prior assurance that this will 
not happen may have little effect in preventing it, I do not feel 
that a formal assurance should be requested. It assumes that the 
.contracting agency is unprofessional. 

Donald Campbell: No! Krf?edcm of information applies to spon- 
sor's freedom too. .3ut the e-aluntor should demand the right to 
publicize his interyu-etation t the same audiences. 

Karl Frey and F.anfred Lang: Misuse of results can only oe ^e- 
grettpd by those who diJ.n't consider the consequences of results in 
a political sy.sttcn yf norms. A reassurance is meaningless and no 
substitute for a though c in advance about possible means to compr.n • 
sate or suppress misleading: interpretations. 

The evaluator should make sure that his results and data are 
not misused. This can be done by explicating^ his design hypotheses, 
samples, methods of assessment and analysis and by giving an unambig- 
uous interpretation. More than this reassurance is rot possible. 

Last Minute Advice to Evaluator'^ 

Question 7: If you were to advise evaluators going into negotiations 
of their i.rst agreement to conduct an evaluation study, 
what would you say to rhem? 

Thomas Owens: Lots of luck! Investigate the government 
bureau's relationship with previous evaluators. Determine why they 
want an evaluation done. Find out the constraints being placed on 
the evaluation. Establish in advance the person(s) to whom the 
evaluator would be resoonsible and the procedures to be followed in 
modifying the contract if necessary. 

David Hamilton: (ij The future career of the evalub.rion rests 
on the 'Outcomes of these negotiations. (2) They should not be 
hurried - keep talking/asking. (3) The negotiations are .just the 
first part of an on,':oing dialogue. If the scen^^ is set properly, 
the actors will be better ablo to play their parrs, (4j Do not 
make promises which you know to be unattainable. Focus on the pos- 
.sible. {'j) Keep thinking and rethinking. 

Robert Kippey: (l) Make certain your agreement touches ail the 
six previous questions. Make sure it is imderstood by all. (2) Make 
certain everyone you will contac': and everyone who is likely to hear 
about your activity kj^ows about you, what your intents are, and how 
to communicate with you. Also make sure you take some initiative 
for touching base with them. (3) Realize that evaluation and change 
can be threatening. Allow ample opportunities for feedback to be 
expressed by all participating. (u) Organize your work carefully. 
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Have clenr statcmet:':.? of .^'"^codures at the outset written so they 
are understood. A-ssi.'-'ri ciear responsibili-^' os. Set time deadlines. 
Describe procedures, m?thodolop;y , instrumentation, and analysis 
early. {3} Make certain that you do not overlook the degree to 
v/hich implementation takes place and the effect which implementation 
has on the feelings of all affected. (6) Spend a lot of time finding 
out what is wanted - what the purpose of the evaluation is and what 
will be helpful to the pro>:?ct, (7) Do not be aloof. Consider 
evaluation as part of the "treatment". For replication, evaluation 
procedures should be replicated as well as treatment. That is, any 
description of treatment should include a description of the evalua- 
tion. Kvaluation does not interact with treatment - it i_s treatment. 
If treatment is "f^oinr-' to hell" and you know v/hy , tell someone. 
Give advice. A lot of times the evaluator simply iinows mor8 than 
anyone else in a pror:ram he is evaluatir-; , at least about the dys- 
f\mctional asnect he has uncovered. 

rJrnest House: The evaluator works with the sponsor. He is not 

* J' 

superior' to h:.n nor is he servile because the sponsor is oaying the 
fee. The e valuator has a resporsibili ty to the sponsor to be as fair 
and as helpful a 5 he can be. He a I do has an obli^^ation to the ulti- 
mate consurr.ers, v;heth?r they bo teachers, parents, or children, to 
be hoiect in his work and careful in it? dissemination, so they will 
come to no harm. This dunl cbli^ptlon iinnoses an ' innenendence on 
uhe evaluator which tnusr be nev;Iy ner^otiatcd v;ithin each setting. 
The evaluator :?ust assume the final responsibility for the integrity 
of his work. 

*Karl Frey and :\anfred Lan;^: Tf an e valuator hrz the ci i:^'ficulties 
formulated in these sever nuestlrns, he should r:iX, first checK his at- 
titudes toward science. L^oes he accent only scientific truth or, in 
addition, normative trutivS. tv^. s he boli e.ve truth finds its way 
without help truth is rienendent on the co.ynltive and emotional 
structure ( Bewusstsein> of humans and of the material conditions in a 
soc iety . ii he accents normative truths a s a reality he should act 
politically, i.e. try to i'ind an-i co-opera;-'^.' with neonle in agreement 
with his ideas, make personnel policy, knov/ th^ rules cf an institu- 
tion and his ^r-r-rr ::ati on and its maintenance. 

Julian Str^r. lie verv carefi'I and thoup.htf ul . Take your time. 
Consult with trusted coll ea rues. Get the full agreement into writing 
and consider that fully. Charr-o -jS hir;h a price as feasible so t-iat 
your renor t is more 1 ik*?? ly to be considered valuable . Be sure tha t 
you v.'ill be c .:rrnensa ted well '"cr all your ef"^'orts, includintT the 
final "clean-'Up". Ho not d:. r. i'ree feasibiiitv or nilct or prelimi- 
narv study. Get oa^?. for it '.oo. because if vou are a true profes- 
sional and insi:-.t on stron.^ saferMarris, you stand an excellent chance 
o 1 no t re 1 1 i n.' the c ^ r; c t . 
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Malcolm Parlett: These negotiations are imporxant. Don't rush 
them, and don't rush into a commitment to do the study. Find out 
why the study is being done; why they have asked you to do it. 
Find out as much as possible about the background circumstances, 
their expectations, and the freedom that they intend to give you. 
Find out whether you are goini^T to have a free hand, or if not, how 
your autonomy is being restricted. Try to get as much in writing 
as possible, perhaps summarising what you see as the decisions made, 
and agreements reached, writing thera up in memorandum form. Send 
this memorandum to them and ask them whether they agree with what 
it contains. There will be some areas, grey areas, where nobody 
can tell in advance what is likely to transpire; new difficulties 
may arise, and ethi>cal problems present themselves. These cannot be 
foretold in detail, but the likelihood of their occurring can be ad- 
judicated, and your likely response to various )iypothetical circum- 
stances can be speculated about. Try and build ar;i understanding re- 
lationship with thera, in such a way that you can come back and nego- 
tiate with them further, in a climate of mutual trust and f orebearance • 
Understand that they have a job to get done?, .sad that they will not 
be able to see all the research dif f icultier aor the pressures upon 
the evaluative researcher, themselves. Try and make clear the types 
of professional dilemma that can arise, tha hesitancies that you 
have, etc. Try and see their point o. ^'iew as much as possible; 
also try to get them to see yours. If you feel uncomfortable with 
the arrangements, and cannot get them clarified to your satisfaction, 
or you don't trust the individuals to keep their side of the in- 
formal bargains made, think twice before finally taking on the 
as2*ignment. If things foul up, and you are pitched headlong into a 
personal/political/ moral morass, the attraction and enjoyment of 
doing the study will be quickly dissipated. If you are flat broke, 
out of work, and desperately need to do the study, you may need to 
take it on while still having reservations. However, do the utmost 
possitio to ensure that there are not contradictory perceptions of 
what you are go in-: to do, and that the Dr obi ems about 'who tnlks to 
whom', when is feedback going to come", etc. are discussed in ad- 
vance of their loomin'r up in unexDected and threatening v/ays. At 
the same time, don't make the contract so rigid that it cannot be 
renegotiated or recast as the study progresses. Build in some 
escape clauses if you discover ; *.at you cannot do the job that you 
said you could. Don't ta'ie on mor-' than you can do - don't get car- 
ried away. Above all, understand that those evaluated, those that 
ycu are responsible to, or those involved, are working for something 
that they believe in, or that they have responsibility for. You have 
a license to enter into tl^eir world. Don^t treat them in any way 
that you would not like yourself be treated, by an outside evalua- 
tor coming into your world. 
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The researchers who answered questionnaires to permit this summary 
were : 



Marvin Alkin 

Center for the Study 

oi Evaluation 
University of California 
Los -Angeles, California 



Ponir'' "l Campbell 
Uepai-x^ment of Psychol O:";;: 
lij.-'thwestem University 



David Cohen 

Center for Educationril 

Policy Research 
Harvard UnW^-rsity 



Karl Frey and Manfred L?n,T 
Institut fUr die Padagofrik der 

Naturv/i s sense ha f ten 
Universitat Kiel 



David Hamilton 
Department of hlducation 
University of Glasgow 



Malcolm Farlett 
Nuffield Foundation 
London, England 



Malcolni Provus 

Evaluation Research Center 

University of Virf^inia 



Robert Rlppey 
Office of Research 

in Health Education 
University of Connecticut 



Robert Stake 

Center for Instructional Respa 

and Curriculum Evaluation 
University of Illinois 



Julian Stanley 

Study of Kathematically 

Precocious Youth 
John Hopkins University 



Wynne Harlen 

Progress in Learninr Science 
University of Read inn 



Dan Stufflebeam 
Center for Evaluation 
V/estern Michi.p:an University 



Ernest House 

Ceuter for Instruction?! Research 

and Curriculum Evaluation 
University of Illinois 



Murray Levine 
Department of Psychol or j 
State University of New 
Buffalo » Mew York 



Yorl: 



Alan Thomas 

Department of Education 
University of Chicaf^o 



Raloh Tyler 

Center for Advanced Study 

Palo Alto 

California 



Thomas Owens 
Northwest Kegional 

Labor'a tory 
Portland, Oreron 



=:duc-?. tional 
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'S. THRE-K ri^-pOTHETICAL COIIVERSATIONS 



The followin{: dialoirue:: 1.llu£5trate the difficulties a commis- 
sioner and a prospective evaluator have in ,r;ettinp" acauointed with 
what the other person needs and expects . Three were written rather 
than one to show how evalnators of different ners\iasionx rccDond. 
There is obviously corrunon cor^cern :^::i<-in'-, thc^io tiii^ee e valuators, but 
clear differences as we i 1 . 

The fir.st evalurjtor rtros.ses the need for maximum nttent--ion to 
rec-ilts that are directly related to the instruction. Thf-' .second 
evaluator stresses; finding out the problems XY,r.z most concern the 
people involved in this particular prorrai;i.T.e . The third evaluator 
stresses the need to remain J ridenendent oC rronsors and nro^^rainne 
personnel. These three evaluators represent the approaches in the 
p;rid in Section III-.' that were called Student Gain by Te^tin,^, 
Transaction-Observation, and Goal-I''ree £>/aluation Approaches. It 
is reasonable to expect that the three contracts -uhey would write 
would be quite different, both in terms of what they v.'culd promise 
and in terms of the safer:uards they would set forth. 

Kollowinr the second dialo^^ue is a collection of responses from 
eirht administrators who were in a position to commission such a 
large-scale evaluation stud'. These responses were made after they 
were invited to Cv->nsider the concerns of the commissioner in the 
second dialogue and to note the uniqueness or c:enerality of the 
si tua tion . 

A conversation between a person v;ho will commission rn evaluation 
study and an evaluation specialist favo':rinf^ a consenuence orientation 

C: Thanks for talcin/; the time to see me today. I suspect that your 
teachinf^ schedule at the University keep.'^. you hopping, but I've 
been told thnt you occasionally carry out educational evalua- 
t ions . 

E: That's ti*ue ^ mv normal teachinr load here at the University is 
pretty heavy, but this quarter is abo^t over. Besides, I am 
wor'tiin:": nov; with ^ small frroup of ,r;raduate students in an eval- 
uation seminar, and when I mentioned the por.sibil ity of evalua- 
ting: your district's pro.iect in iieality-P.ooted Readinr they 
became really interested. 

C: You moan you mi.^ht use students in carryinr; out an evaluation? 

E: It's really rood experience for them, and they often cnn make 
excellent contributions to the evaluation itself. Of course, 
one must bo careful not to exploit students in such situa.tions. 
Too many of mv col lea^'uos v : ev ■ raduate students as a somev/hat 
advanced form of mirrant v;>r.-<ers. 



C: Well, did you have a chance to read the vnrite-up I sent you of 
our new Reality-Rooted Reading programme? We think it holds 
great promise as a vay to get poor readers more jrjvolved in de- 
veloping their reading skills. 

E: I did read the document, and you may be correct. There are cer- 
tainly a number of positive features the prop;ra:nme. I must 
confess, though, that- I was disturbed by the apparent lack of 
replicabili ty in the pro^rramme itself. It sounds more like a 
six ring circus than anythin;'. which, if it does work, could be 
used again in the future. If you're going to the trouble of 
evaluating this intervention, I assuiro that you're contemplating 
its use in the future. Interventions that are not at least 
somewhat replicable can't really be emDloyed very well in the 
future. Is your Reality-Rooted Reading programme goin^ to be 
essentially reproducible? 

C: I'm glad you brought that up. The planning committee which has 
been working out the p.^ogramine ' s details became aware of that 
problem a i^v/ weeks c\f;r>. They're in the process of devising 
instructional guides which will substantially increase the re- 
plicability of the programme. 

E: I just hope the planning committee itself is rooted in reaT.ity. 

C: Well, what about the evaluation? Will you take it on? Our 
district' school board is demanding formal evaluations of all 
new Drop-rarumes such as this one, so we can't really get under 
way until the responsibility for evaluation has been assigned. 

E: I'll need to get some questions answered first. 

C: Fire away. 

E: 'that's the purpose of the evaluation? In other v/ords what's 

going to happen as a consequence of the evaluation? Unless the 
evaluation is .Toinf/; to make a genuine difference in the nature 
of the instructional prof^ramme, v;e wouldn't want to muck with 
it. Too many of us here at the University have ex^jerienced the 
frustrations of carrying out research studies whose only pur- 
pose seemed to be that of widening the bindings of research 
journals. Unless an cv'=>luation satisfies the "so what?" cri- 
terion, I'm sure *.'e wouldn't V'^ interested. 

C: Well, the- d\:.vtrict superintendent has ir.dicated that the con- 
tinuation of the new programme v;ill tt:? totally d'3pendent upon 
the rcsMlt^- of its evaluation. That sitisfy you? 

E: Sure does. :iow, t.u.re was a :-j t of rhe-^oric in your programme 
description about appraising xne i; rc-.Tr.nnne in terms of the 
"uniquentrss of ' ts mnovntive feptuicr''. Does that imply you're 
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more concerned with evaluating the procedural aspects of the 
pro^^ramme than with evaluating the results yielded by those 
Drocedures? This ir. =i particularly important issue for me. 

V/ell, we are very proud of the proj^ramme's new features. V.hat 
are you ^retting at? 

There are too mnnv educators who are so cau»-ht up with the rap- 
tures of an instructional Innovation that they are almost ob- 
livious of its effects on learners. And that, after all, is 
why we're in the i^ame. Our instructional interventions should 
help learners. I want to be sure that, although we will con- 
sider the r^rocedures employed durir : the programme, the main 
emnhai^is of the ovalu^ition will focus on the consequences of 
that Drof?;ramme ' s use. 

Oh, we'd be oerfectly arreeable to that. After all. you neople 
are the exnerts. Besider, I fruesr. I share your point of view. 

I also noted an nlrnos^ exclusive nreoccupation with co/rni -uive , 
that is, intellectual outcomes of the nro^-^ramme . Your neople 
r-eern^-'d t.o be concerned c.nly nbout the slUlls of readint;. Aren't 
you also worried .'bout: nunlls' attitudes toward reading? 
< r course, but you Cr-n't .or>resr, that kind of stuff can you? I 
thou.'ht the ifrnct.ive domain was off-limits for the kinds of 
evaluators who, -^s vcj appar-ently nre. are concerned with evi - 
den '-o . 

rt's t?n"h to r'o. but there are some reasonably ^ood ways of 
rettlnr evjilenco re-'»rcMn : learTPM-s' affect toward en Instruc- 
ts one I :)vr, •* rMinrne . '.e ' d v/^nt to use Miern. 

: ' w a t e s t r ^ i 1 '1 ;/ 0*1 h a e t bu i 1 d 1 c t s of n e w one s ? 

'■>' • ■'"-hat v/e will have to devlsf? some new measures. The 

star/' r Arer\ tenchinr tests your district now uses will be 
v-T'thiess W^r f \.i s k i rul or an eva lu;i t i . Wo'll need to see if 
cho-f. pf.> .-iiy nvailnble crlter : c n-reforenced tests v/hi ch we can 

U S ! ' T"' -I '-^ I" . 

'.y') you - ' ' 7f ] v.Tjvr ve use te*: tsV 

llo.but it is inooT'^an!: to -o l riirricient evidence re^^ardWi/: a oro- 
.•rann" ' - i-rtt^clr r.'^. th.';t. v."* PVf> in c better oosition to anoraise 
.1 tr r"n.-eo^f 'y.i: r.K.MM ::i»~'r( - i." oy In tu i t i ri.-- th f r,onsen.uencr»s . 
■.'■■•■*!' r\\'\ li-.".. »: :;;■>.. .i-d 'r:^r>?:rr, vn'r, you? 
'>r-np^ : :i 1 V . .;'id:'ir.''r' t r r?';:r.ri on eviiieno^- lonri tn bo betrter 

then ,i'i,l;-fT[f»ri tr :.:'<\n v; i t.h'M.; t; it.. I ro:i»:Tl7 devi s-ed ne^surinr de- 
vices -an .:'ten be hflr-.Tn: in detMcnn/- a ivrorrnmnie ' r effects, 
bi^th th :'re Lr.o f wmw^ i:.t'-':]d"i ^t. vk- 1. 1 '-^s ,inv unan t lei na ted 
effect:-. 



How come T haven't heard you say "instruc tional objectives" 
once during our conversation? T thought you folks were all 
strung out on behavioural objectives. 

Well, clearly stated instructional objectives represent a use- 
ful way of describing a programmers intended effects. But the 
effects of the programme are what we want to attend to, not J^st 
the educator »s utteranceti about what was supposed to happen. 
Consequence-oriented educational evaluators can function effec- 
tively even without behavioural objectives. 

Amazing! 

There are a couple of other areas we have to get into. I hope 
you're sincere In wanting to contrast the new programme ' with 
alternative ways that the money it^s costing might be spent. 

Absolutely. 

And, finally, the matter of evaluator indepenuence . Will we 
have the right to release the results of our pvaluation to all 
relevant decision-makers involved in this project, including 
the public? 

You think that's important to get clarified now? 

It might head off some sticky problems later. We M like that 

kind of independence. 

I think It can be assured. I»ll want to check it out with my 
division 'chief, however; 

There's also a related kind of independence I want to discuss. 
V'.like some of the independent evaluation firms thot hflve sprung 
up in the past few years, we really aren»t in the evaluation 
business on a full-time basis, hence iii a sense we don»t need 
your district's repeat business. Therefore, we'll be inclined 
to call our shots openly, even if it means that the programmp 
is evaluated adversely. 

That's related to your earlier point about Independence in re- 
porting the evaluation's results. 

You bet. 

Okay, we're v.'illlng to play by the rules. T hope it turns out 
positively though. \ 

So do I. Our kldt; couJd surely do with a bit of help in their 
reeding programme. 
Well, what next? 

Why don't I and some of my students whip up a detailed plan of 
how we want to do the evaluation and fire it off to you by mail, 
say, in two weeks. 
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C: Fine, If we have any problems with it, we can get back to you. 
All right? 

E: Sure, 

C: We haven *t talked about money yet. How much will this thing 
cost? 

E: We'll include a budget with our evaluation plan. But, because 
university professors are so handsomely rewarded by their own 
institutions, I»m sure the amount will be a pittance, perhaps 
a used chalkboard eraser or two, 

C: You guys do live in an invory tower, don*t you? 

E: Didn*t you take the elevator on the way up? 



A conversation b etween a person who will commission an evaluation 
study and an evaluation specialist favouring a responsive approach 

C: As I said in my letter I have asked you to stop by because we 
need an evaluator for our National Experimental Teaching Pro- 
gramme. You have been recommended very highly. But I know you 
are very busy . 

E: I was pleased to come in. The new Programme is based on some 
interesting ideas and I hope that many teachers will benefit 
from your work. Whether or not I personally can and should be 
involved remains to be seen. Let's not rule out the possibili- 
ty. There might be reasons for me to set aside other obliga- 
tions to be of help here. 

C: Excellent. Did you have a chance to look over the programme 
ma terials I sent you? 

E: Yes, and by coincidence, I talked with one of your field super- 
visors, Mrs. Eater,. We met at a party last week. She is quite 
enthusiastic about the plans for group problem-solving activities 
That is one thing we need evaluation help with. What kind of 
instruments are available to assess problem-solving? Given the 
budget we have, should we try to develop our own tests? 

E: Perhaps so. It Is too early for me to tell. I do not know 

enough about the situation. One thing I like to do is to £,et 
quite familiar with the teaching and learning situations, and 
with what other people want to know, before choosing tests or 
developing new ones. Sometimes it turns out that we cannot 
afford or cannot expect to get useful information from student 
performance measures. 



C: But surely we shall need to provide some kind of proof that the 
students are learning more, or are understanding better, than 
they did before! Otherwise how can we prove the change is 
worthwhile? We do have obligations to evaluate this programme. 

E: Perhaps you should tell me a little about those obligations. 

C: :"'-,Yes. Well, as you know, we are under some pressure from the 
Secretary (of Health, Education and Welfare), from Members of 
Congress, and the newspapers. They have been calling for a 
documentation of "results". 

But ^U3t as important, we in this office want to know what our 
programme is accomplishing. We feel we cannot make the best 
decisions on the amount of feedback we have been getting. 

E: Are there other audiences for information about the National 
Experimental Teaching Programme? 

C: We expect others to be interested. 

E: , Is it reasonable to conclude that these different "audiences" 

will differ In what they consider important questions, and per- 
haps even what they would consider credible evidence? 

C: Yes, the researchers will want rigor, the politicians will want 
evidence that the costs can be reduced, and the parents of stu- 
dents will want to know it helps their children on the College 
Board Examinations. I think they would agree that it takes a 
person of your expertise to do the evaluation. 

E: And I will look to them, and other important constituencies, 

teachers and taxpayers, for example, to help identify pressing 
concerns and to choose kinds of evidence to gather. 

C: Do you anticipate we are going to have trouble? 

E: Of course, I anticipate some problems in the pi'ogramme. I 
think the evaluator should check out the concerns that key 
people have. 

C: I think we must try to avoid personalities and stick to objec- 
tive datf^. 

E; Yes, I agree. And shouldn*t we find out which data will be 
considered r'elevant to people who care about this programme. 
And some of the most important facts may be facts about the 
problems people are having with the programme. Sometimes it 
does get personal. 

C: The personal problems are not our business. It is important 
to stick to the Impersonal, the "hard-headed" questions, like 
"How much is it costing?" and "How much are the students learn- 
ing?" 



To answer those questions effectively I believe we must study 
the programme, and the communities, and the decision-makers who 
will get our information. I want any evaluation study I work 
on to be useful. And I do not know ahead of time that the cost 
and achievement information I could gather would be useful. 
I think we know what the funding agencies want: information on 
cost and effect. 

We could give them simple statements of cost, and ignore such 
costs as extra work, lower morale, and opportunity costs. We 
could give them gain scores on tests, and ignore what the tests 
do not measure. We know that cost and effect information is 
often superficial, sometimes even misleading. I think we have 
an obligation to describe the complexities of the programme, in- 
cluding what it is costing and what its results appear to be. 
And I think we have an obligation tc say that we cannot measure 
these important things as well as people think we can. 
Well, surely you can be a little less vague as to what you 
would do. We have been asked to present an evaluation design 
by a week from next Wednesday. And if we are going to have any 
pretesting this year we need to get at it next moath. 
I am not trying to be evasive. I prefer gradually developed 
plans - "progressive focusing" Parlett and Hamilton call it. I 
would not feel pressed by the deadline. I would perhaps pre- 
sent a sketch like this one (drawing some papers from a folder); 
one which Les McLean used in the evaluation of an instant-access 
film facility. His early emphasis was on finding out what issues 
most concern the people in and around the project. 
I think of that as the Programme Director's Job. 
Yes, and the evaluation study might be thought of - in part - 
as helping the Programme Director with his Job. 
Hmmm. It is the Secretary I was thinking we would be helping. 
You made the point that different people need different infor- 
mation, but it seems to me that you are avoiding the information 
that the Secretary and many other people want. 
Let's talk a bit about what the Secretary, or any responsible 
official, wants. I am not going to presume that a cost- 
effectiveness ratio is what he wants, or what he would find use- 
ful. We may deciJe later that it is. 

First of all, I think that what a responsible official wants in 
this .situation is evidence that the National Programme people 
are carrying out their contract, chaz the responsibility for 
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developing new teaching techniques continues to be well placed, 
and that objectionable departures from the norms of professional 
work are not occurring. 

Second, I think a responsible official wants information that 
can be used in discussions about policy and tactics. 
Our evaluation methodology is not refined enough to give cost- 
effectiveness statements that policy-setters or managers can 
use. The conditionality of our ratios and our projections is 
formidable. What we can'do is acquaint decision-makers with 
this particular programme, with its activities and its statis- 
tics, in a way that permits them to relate it to their experi- 
ences with other programmes. We do not have the competence to 
manage educational programmes by ratios and projections 

- management is still an art. Maybe it should remain an art 

- but Tor the time being we must accept it as a highly partic- 
ular i rod and judgmental art. 

I agree - in part. Many evaluation studies are too enormously 
detailed for effective use by decision-makers. Many of the 
variables they use-^^re simplistic, even though they show us how 
their variables correlate with less simplistic measures. Some 
studies ignore the unrealistic arrangements that are made as 
experimental controls. But those objectionable features do not 
make it right to de-emphasize measurement. The fact that 
management is an art does not mean that managers should avoid 
•good technical information. 

What I want from an evaluation is a good reading - using the 
best techniques available - a good reading of the principal 
costs and of the principal benefits. I have no doubt that the 
evaluation methodology we have now is sufficient for us to show 
people in government, in the schools, and in the general public 
what the programme has accomplished. 

If I vere to be your evaluator I would get you that reading. I 
would use the best measures of resource allocation, and of 
teaching effort, and of student problem-solving we can find. 
But I would be honest in reporting the limitations of those 
measures. And I would find other ways also of observing and 
reporting the accomplishments and the problems of the National 
Programme . 

That of course Is fajr. I do not want to avoid whatever real 
problems there may be. I do want to avoid collecting opinions 
as to what problems (and accomplishments) there might be. I 
want good data. I want neither balderdaiJh nor gossip. I wa. t 
my questions answered and I want the Secretary »s questions 
answered. 
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And those questions might change as we go along. You would 
call that "formative evaluation"? 

E: Sometimes, I would also call it "responsive". 

C: What kind of final report would you prepar'e for us? 

E: I brought along a couple of examples of previous reports. I 
can leave them with you. I can provide other examples if you 
would like. 'Aether there is a comprehensive or brief fiOQl 
report, whether there is one or several, those decisions can 
be made later. 

C: No, T»m a;.raid that simply won»t do. If we are to commit funds 
to an evaluation study, we must have a clear idea in advance of 
how 'ong it is going to take, what it will cost, and what kind 
of product to expect. That does not mean that we could not 
change our agreement larer. 

E: If you need a promise at the outset, wo can make it. Believe 
me, I do not believe ^t is in your best interests to put a lot 
of Rpecif icatioris inro the "contract". I would urge you to 
choose your evaluator in terms of how well he has satisfied hi3 
previous clients more than on the promises he would make so 
*?arly . 

C: I ^. would be irresponsible of me not to have a commitment from 

E: Of course. And your evaluator should take SDme vi the in! ■:ia- 
:ive 3n proposing what should be specified and what options 
jhoula be left open. 

i.' ; Let me be frank about one worry T have. I am afr^^id I may get 
an evaluator who is going to use our funding to "piggy-bnck" 
a research project he has be'v; wanting to do. He might a^reo 
to do "ou:*" evaluation study but it might hav^ very little to 
do with the key enrols lens of the Experimental Teaching ^ro- 
gr^mme . 

E: It is reason?. e to expect any investigatoi to continue old 
interests in ^^ew ?:arroundings . When you buy him you buy his 
curiosities. ::e .ay develop hypotheses, for example, about 
problem so\v''\g d ^. caching sty*'--, h^notheses that sound most 
relevant to • nf.- r.ro- 'ciFTi^.e - but '.i,, looL of these hypotheses 
may be of litxiu u.^e +vj those v.ni sponsor, op«?rate, or benefit 
from I h ' • p ' > fv; r a ra m e . 

His f /ru • i 1-e tactics, a carefully controlled comparative re- 
sea rc^i ' I'^'ort or a historical longitudinal research study, for 
example, might be attractive to your .staff. But he is not 
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inclined to talk about how unnecessary this approach may be. 
The inertia (n his past work may be too strong. You are right, 
there is a danger. I think it can best be handled by looking 
at the assignments the evaluator has had before, and by getting 
him to say carefully what he is doing and why, and by the spon- 
sor saying very carefully which he wants and does not want, and 
by everybody being sceptical as to the value of each under- ^ 
taking, and suggesting alternatives. 

Would you anticipate publishing the evaluation study in a pro- 
fessional journal? 

Even when en article or book is desired it is rare for an eval- 
uation study to be suitable for the professional, market. 
Eval t^-.n studies are too long, too multi-purposive, too non- 
gem raVzable and too dull for most editors. Research activi- 
ties within the evaluation project sometimes are suitable for 
;,: :iudience of researchers. 

r usually suppose that my evaluation vrk is not done for that 
rarpose. If something worth publishing, became apparent I would 
talk over the possibilities with you. 

I think something like that should be in writing. What other 
assurances can you give me that you would not take advantage 
o£ us? Do you operate with some specific "rules of confiden- 
tiality"? 

I would have no objection to a contract saying that I would not 
release findings about the project without your authorization. 
I consider the teachers, administrators, parents and children 
also have rights here. Sometimes I will want to get a formal 
release from them. Sometimes I will rely on my judgment as to 
what should and should not be made public, or even passed along 
to you. In most regards I would follow your wishes. If I 
should find that you are a scoundrel, and it is re.-.^nt to my 
evaluation studies. I will break my contract and pass the word 
along to those whom I believe should know. 

I have nothing to lose, but others involved may have. I do not 
^ant to sanction scurrilous muck-raking in the name of indepen- 
dent evaluation. I wonder if you are too ready to depend on 
your own judgment. What if it is you who are the scoundrel? 

I would expect you to expose me. 
. By exposing you I would be exposing my bad judgment in selecting 
you - the line of thought I would return to is the safeguard 
you wouM offer us against mismanagement of the evaluation study. 
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The main safeguard. I think, is what I was offering at the be- 
ginning: communication and negotiation/ In day to day matters'" 
I make many decisions, but not alone. My colleagues, my spon- 
sors, my infonnation sources help make those decisions. A good 
contract helps, but it should leave room for new responsibili- 
ties to be exercised. It should help assure us that we will 
get together frequently and talk about what the evaluation 
study is doing and what it should be doing. 

What about your quickness to look for problems in the programme? 
Perhaps you consider your own Judgment a bit too precious. 
I do not think so. Perhaps, I try to get confirmation from 
those I work with and from those who see things very differently 
than. I do. I deliberately look for disconf irmation of the Judg- 
ments I make and the Judgments I gather from others. If you 
are thinking about the Judgments of what is bad teaching and 
learning I try to gather the Judgments of people both who are 
more expert than I and those who have a greater stake in it 
than I. I cannot help but show some of my Judgments, but I 
will look for hard data that support my Judgment and I will 
look Just as hard for evidence that runs counter to my opinion. 
That was nicely said. I did not mean to be rude. 
You speak of a problem that cuts deeply. There are few depend- 
able checks on an eva^:uator's Judgment. I recognise that. 

You would use consultation with the project staff and with me, 
as a form of check and balance. 

Yes. And I think that you would feel assured by the demands I 
place upon myself for corroboration and cross-examination of 
findings. 

Well, there seems to me to be a gap in the middle. You have 
talked about how we would look for problems and how you would 
treat findings - but will there be any findings? What will the 
study yield? 

If I were to be your evaluator we might start by identifying 
some of the key aims, issues, arrangements, activities, people, 
etc. We would ask ourselves what decisions are forthcoming, 
what information would we like to have. I would check these 
ideas with thp programme staff. I would ask you and them to 
look over some thingy I and other evaluators hfl\ e done in the 
past, and say what Looks worth doing. The problem would soon 
be too big e muddle, and we would h^ve to start our diet. 
I don^t cmre much for the metaphor. 



E: That may be as good a basis as any for rejecting an evaluator 

- his bad choice of metaphors. 
C: I've just realized how late it is. I am hoping not to be re- 
jecting any evaluators today. Perhaps ycu would be willing to 
continue this later. 
E, Let me make a proposal. I appreciate the immediacy of the 

situation. I know a young woman with a doctorate and research 
■ exT.erience, who might be available to co-ordinate the evalua- 
tion work. If so, I could probably be persuaded to be the 
director, on a quarter-time basis. Let me go over your mate- 
' rials with her. We would prepare a, sketch of an evaluation 
plan, and show it to you along vitb some examples of her pre- 
vious work. 

C: That is a nice offer. Let me look at your examples and think 
about it before you go ahead. Would it be all right if I 
called you first thing tomorrow morning? Good. Thanks very 
much for coming by. 

Commentary - rnnversation with a Responsive Evaluator 

I recognised as I started to develop this conversation that the 
negotiating conditions tnat I had experienced and the rhetoric I was 
familiar with were not common in other countries. I felt that the 
tenacious reader could overcome those parochial features to get at 
issues that are common to evaluation of large-scale programmes any- 
where. 

Many .aders would prefer a listing of issues rather than a dia- 
logue; 1 agree that listings can be helpful. I do believe that 
issues take on a different meaning vl.en they are presented in natu- 
ral discourse, and that it is useful for practitioners and theorists 
alike to give attention to these different meanings. The inter- 
weaving of pride, vulnerability, aspiration, and other personal and 
political characteristics into educational purpose and method are 
more apparent in such discourse than in such a checklist as I pre- 
sented earlier. 

I wanted to keep it a two-person, informal situation for simpli- 
city and because I guessed that that would be most common in Europe. 
In the U.S.A. open bidding for evaluation contracts is required by 
law for many national and state programmes. .The negotiations have 
become formal, legalistic, often impersonal, with little attention 
to the issues raised in this dialogue. Since I was too unfamiliar 
with any European setting and frame of mind I kept the idea of a 
conversation but made it an American scenario. 
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I sent an early draft of this conversation to the following 
persons for comments, particularly as to how the Commissioner might 
respond: 



Heinrich Bauersfeld r a Becher 



Astrid NystrOm University of Pittsburgh 

S?ockSolm^°'"'''°^ Education Lawrence Stenhouse. Director 



Centre for Applied Research in 
Kn-, ^ , « Education 
Executive llrellTr' University of East Anglia 

Stiftelsen Riksbankens Marc Tucker 

Stockh^!r''°"' ^/^ir'l °^ Education 

Washington, D.C. 

Most of these respondents found the issues rele-ent and difficult to 
resolve. Most noted the discrepancies between their real situations 
and the fJctltlous situation. I used many of the wordings they sug- 
gested in revising the dialogue. Most of them wanted a quicker 
declaration of -purposes and plans, but my responsive evaluator is 
convinced that many of the shortcomings of evsluation studies are 
traced to a willingness to guess at what the key variables and issues 
are and to make an irretrievable commitment to personnel and instru- 
mentation. And so he is vague, but explorative: anxious to base 
his worthiness on past performance rather than on what he might 
promise at present. It is not a stance that ou the respondents 
found persuasive. Some of the reactions of the eight administrators'" 
named above are most insightful. I quote a sample of these reactions: 
"Now. if this were my country the project leader would probably 
not be so free to make his own decisions. They would have been 
made at a 'higher' level." 

"Over here a project director has to fight to get his project 
evaluated. " 

"...how clearly this conversation brought out the differing 
assumptions made m the U.S. from those which would be made in 
this count-y - for example that funds ere normally made avail- ■ 
ab-.e under c-ntract, that such contracts stdpulate the need for 
evaluation, and that tr.^y impose tight deadlines." 
"The dialogue presupposes that the evaluator is a well-Known 
person,... A minor researcher, on his way up.... might expect to 
be treated with more firmness and prejudice bv the commissioner." 



"I read your hypothetical conversation over and over again, and 
(I found the ©valuator) too damn patronizing. " 

•It is necessary to clarify the project philosophy. What really 
shall be imDroved? ...the teaching of fractions, the sensitivi- 
ty of the teacher, the degree of variety/differentiatedness, 
the style of interaction between teacher and students, or what 
else?" 

"It should be more ^product ' -oriented , a typical concern of 
action administrators." 

"The commissioner might probe more to find out if the evaluator 
is really sympathetic to the programme, shares the values of 
the creators, thinks it has promise, is willing to concentrate 
on the goals and variables important to the project staff, 
shares with the project staff some agreement as to what might 
be valid indicators of project success." 

"Many commissioners would (want) the evaluator to present re- 
sults ... vMoh go along with (their) own interests, or those of 
the decision makers at higher levels. The Commissioner might 
discard the evaluator as soon as he realizes that the desired 
results cannot be bought so easily." 

"I would have had the commissioner more tough-minded. . .raising 
questions about time span, costs, and scope of the evaluation; 
the experience and qualifications of The evaluator; the 
•subjectivity' of' the proposed evidence, the apparent untidi- 
ness of the -evaluation design, and whether the evaluator's re- 
port will offer recommendations about the continuance or dis- 
continuance of the experiment," 

«The commissioner might press for what the evaluator means by . 
'usefulness' .. .is it the furtherance of science, of psycho- 
logic.-,! methods, of parent insight, of the manipulative power 
of steering board members, of the child's self awareness, 
of...? If the evaluator tries to serve all these needs, he is 
overburdened and soon will become a distractor and an explosive 
power in the project." 

"The pDint is: How much do the coainl.^sl oner and evaluator 
Join the common insincority of producing designs ^o which they 
do not stand by?" 

"The conmlssijgner might veil want to knov ^ow much time of how 
many people with what kind of expert en-e and training will be 
required, and at v^'iat ccst." 



"Recognition should be made of the fact that commissioners do 
have particular constituencies and that a wise evaluation would 
help meet the questions of these constituencies, at the same 
time getting on with the problems that may lie deeper." 

"The commissioner might want to know whether the evaluator*s 
preliminary observations will be available to project staff as 
the project proceeds, whether such views will be available to 
others, and what control he will have over premature release of 
findings . " 

"The commissioner lairs-it want some * right of reply* to negative 
findings, whenever they are released, or a chance to confer 
with the evaluator on the basis of draft reports before the 
final reports are published." 

"The commissioner uight want to know how disruptive the evalua- 
tor will be, how much time of project staff he will take, whether 
he can make use oi measures routinely administered, whether he 
-Will" demand tha use of control groups, etc." 

"If the evaluator *s racial, ethnic, or cultural background are 
different froD staff and students, the commissioner might try 
to find out if the evaluator sees those differences as a prob- 
lem, and, if so, what the evaluator would propose to do about 
it." 

"Is it not too egotistical to * learn as much as we can fj'om the 
project*? Such an outcome is the byproduct of the project, too 
often the only worthwhile one. A foundation will hardly pay 
million:, for the learnings of the project staff and evaluators." 

"I would have suggested some' additional issues, such as that 
of access "to participating schools; information about the proj- 
ect's underlying aims: the right to corf identia lity of par- 
ticipants in the programme, and the concern for *proof * , rather 
than evidence, of success or failure." 

"I don't think our evaluators have audiences. They have to 
create them . " 

Most of these respondents felt that more should have been accom- 
plished in that hypothetical meeting of commissioner and evaluator. 
Most t.iought that it wns realistic and included some of the more im- 
portant issues that ne^^d to be resolved in such a negotiation. 



A c.pnv?'nv.tlon between a person who Is coaunlaslonlnp; an Independent 
evaluation study and the evaluator who favours a "go al-free" approach 

C: Well, we're very glad you were able to take this on for us. We 
consider thit: programme in. reading for the disadvantaged to be 
one of the most important we have ever funded. I expect you'd 
like to get together with the project staff as soon as possible 
- the director is here now - and of course, there's quite a 
collection of documents covering the background of the project 
that you'll need. We've assembled a set of these for you to 
take back with you tonight. 
E: Thanks, but I think I'll pass on meeting the staff and on the 
materials. I will have my secretary get in touch with the 
director soon, though, if you can give me the phone numbers. 
C: Y J icdn you're planning to see them l ater ? But you've got so 
little time - we thought that bringing the director in would 
really speed things up. Maybe you'd better see him - I'm 
afraid he'll be pretty upset about making the trip for nothing. 
Besides, he's understandably nervous about the whole evaluation. 
1 think his team is worried that you won't really appreciate 
their approach unless you spend a good deal of time with them. 

E: Unfortunately, I can't both evaluate their achievements with 
reasonable objectivity and also go through a lengthy indoc- 
trination session with them. 

C: Well, surely you want to know what they are trying to do - what's 
distinctive about their approach? 

E: I already know more than I need to know about their goals 
- teaching reading to disadvantaged youngsters, right? 

C: But that's so vague - why, they developed their own instruments, 
and a very detailed curric You can't cut yourself off 

from that ! Otherwise, you'll finish up criticizing them for 
failing to do wh^t they never tried to do. I can't let you do 
that. In fact, I'm getting a little norvous aboux Letting you 
go any further with the whole thing. Aren't you going to see 
them at all ? You're prrposing to evaluate a three million dol- 
lar project without evei ' ooking at it? 

E: As far as possible, yes. Of course, I'm handicapped by being 
brought in so l.ite and under a right deadline, so I may have 
to make s ■ re compromises. On the general issue, I think you're 
suffering from some misconception about evaluation. You're 
usea to the rather cozy re:ationship which often - in my view - 
contaminates the objectivity of the evaluator. You * ulc3 think 
about the evaluation of drugs by the double-blind approach... 
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C: But even there, the evaluator has to know the intended effect 
of the drug in order to set up the tests. In the educational 
field, it's much harder to pin down goals and that's where 
you'll have to get together with the developers. 

E: The drug evaluator and the educational evaluator do not even 
have to know the direction of the intended effect, stated in 
very general terms, let alone the intended extent of success. 
It's the evaluator's job to find out what effects the drug has, 
and to assess them. If (s)he is told in which direction to 
look, that's a handy hint but it's potentially prejudicial. 
One of the evaluator's most 'JSeT.il contributions may be to 
reconceptualize the ef^fectr. i-.ittier than regurgitating the ex- 
perimenter' s conception of ' it'^ti. . 

C: This is too far-out altoge ler ' t\at are you suggesting the 

evaluator do - test for effex: i ^n every possible variable? He 
can't do that. 

E: Oh, but he has to do that anyway > I'm not adding to his burden. 
How do you suppose he picks up side effects? Asks the experi- 
menter for a list? That would be cozy. It's the evaluator's 
job to look out for effects the experimenter (or producer etc.) 
did not expect or notice . . The so-called "side effects", whether 
good or bad, often wholly determine the outcome of the evalua- 
tion. It's absolutely irrelevant to the evaluator whether these 
are "side" or "main" effects; that lang.-tdge refers to the 
intentions of the producer and the evaluator isn't evaluating 
intentions but achievements . In fact, it's risky to hear even 
general descriptions of the intentions, because it focuses your 
attention away from the "side-effects" and tends to make you 
overlook or down weight them. 

C: You still haven't answered the practical question. You can't 

test for all possible effects. So this posture is absurd. It's 
much more useful to te]l the producer how well he's achieved 
what he set out to achieve. 

E: The producer undoubtedly set out to do something really worth- 
while in education. That's the really significant formulation 
of his goals and it's to that formulation the evaluator must 
address himself. There's also a highly particularir-ed descrip- 
tion of the goals - or there should be - and the producer may 
need some technical help in deciding whetner he got there, but 
that certainly isn't what you , as the dispenser of taxpayer's 
funds, need tc know. You need to know f • the money was wasted 
or well-spent etc. 
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Look, I already had advice on the goals. That»s what my advi- 
sory panel tells ne when it recommends which proposal to fund. 
What I'm paying X£ii ^oi* to .1udge success, not legitimacy of 
the direction of effort. 

Unfortunately for that way of d.Widing the pie, your panel can't 
tell what configuration of actual effects would result, and 
that*s what I'm here to assess. Moreover, your' panel is Just 
part of the whole process that led to this product. They're 
not immune to criticism, nor are you, and nor is the producer. 
(And nor am I.) Right now, you have - with assistance - 
produced something, and I am going to try^to determine whether 
it has any merit. When I've produced my -evaluation, you can 
switch roles and evaluate it - or get sc^eone else to do so. 
But it's neither possible nor proper for an evaluator to get by 
without assessing the merits of what ha^s been done, not Just 
itc consonance with what someone else thought was meritorious. 
It isn't proper because it's passing the buck, dodging the - or 
one of the - issue(s). It isn't possible because (it's almost 
certain that) no one else has laid down the merits of what has 
actually happened. It's very unlikely, you'll agree, that the 
producer has achieved exactly the original goals, without short- 
fall, overrun or side-effects. So - unless you want to abrogate 
the contract we Just signed - you really have to face the fact 
that I shall be passing on the merits of whatever has been done 

- as well as determining exactly what that is. 

I'm thinking of at least getting someone else in to do it too 

- someone with a less peculiar notion of evaluation. 

*I certc^.i.:- hope you do. There's very little evidence about 
the ir.ueroudge reliability of evaluators. I would of course co- 
operate fully in any such arrangement by refraining from any 
communication whatsoever with the other evaluator. 
I'm beginning to get the feeling you get paid rather well for 
speaking to no one. Will" you kindly explain how you're going 
to check on all variables? Or are you going to take advantage 
of the fact that I have told you it's a reading programme - I'm 
beginning to feel that I let slip some classified information. 
What's your idea of an ideal evaluation situation - one where 
you don't know what you're evaluating? 

In evaluation, blind is beautiful. Remember that Justice her- 
self is blind, and good medical research is double blind. The 
educational evaluator is severely handicapped by the impossi- 
bility of double-blind conditions in most educational contexts. 
But (s)he must still work very hard at keeping out prejudicial 



information. You can't do an evaluation without know'.ng what 
it is you're supposed to evaluate - the treatment - but you do 
not need or want to know what it's supposed to do. You've al- 
ready told me too much in that direction. I still need to know 
some things about the nature of the treatment itself, and I'll 
find those out from the director, via my secretary, who can fil- 
ter out surplus data on intentions etc. before relaying it to 
me. That data on the treatment is what cuts the problem down 
to size: I have the knowledge about probable or possible ef- 
fects of treatments like that, from the research literature, 
that enables me to avoid the necessity for examining all pos- 
sible variants. 

C: Given the weakness of research in this area, aren't you still 
pretty vulnerable to missing an unprecedented effect? 

E: Soniewhat, but I have a series of procedures for picking these 
up, from participant observation to teacher interview to samp- 
ling from a list of educational variables. I don't doubt I 
slip up, too; but I'm willing to bet I miss less than anyone 
sloshing through the swamp towards goal-achievement. I really 
think you should hire someone else to do it independently. 

C: We really don't have the budget for it... maybe you can do some- 
thing y*;ur way. But I don't know how I'm going to reassure the 
project staff. This is going to seem a very alien, threatening 
kind of approach to them, I'm afraid. j 

E: P??cple that feel threatened by referees who won't accept their 
hospitality don't understand about impartiality. This isn't 
support for the enemy, it's neutrality. I don't want to penal- 
ize them for failing to reach over-ambitious goals. I want to 
give them credit for doing something worthwhile In getting half- 
way to those goals. I don't want to restrict them to credit for 
their announced contracts. Educators often do more good in unex- 
pected dir*. tions than the intended ones. My approach preserves 
their chance in those directions. In my experience, interviews 
with project staff are excessively concerned with explanations 
of shortfall. But shortfall has no significance for me at all. 
It has some for you, because it's a measure of the reliability 
of the projections they make in the future. If T were evaluating 
them as a production team, I'd look at that as part of the 'Crack 
record. But right now I'm eva'luating their product - a reading 
programme. And it may be the best in the world even if it's 
only half as good as they intended. No, I'm not wor.: . ng in a 
way that * s prejudiced aga inst them. 
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rm still haunted by a feeling this is an unrealistic approach. 
For example » how the devil would I ever know who to get as an 
evaluator except in terms of goal-loaded descriptions. I got 
you - in fact, I invited you on the phono - to handle a "reading 
programme for disadvantaged kids" which is goal-loaded. I 
couldn't even have worked out whether you'd had any experience 
in this area except by usin^^ that description. Do you think 
evaluators should be universal geniuses? How can they avoid 
goal-laden lang\iage in describing themselves? 

There's nothing wrong with classifying evaluators by their oast 
performance. You only risk contamination when you tell them 
what you want them to do this time, using the goals of this pro- 
ject as you do so. There's nothing unrealistic abouL the al- 
ternative, any more than there is about cutting names off 
scientific papers when you, as an editor, send them out to be 
refereed. You could perfectly well have asked me if I was free 
to take on an evaluation task in an area of previous experience 
- a particularly important one, you could have added - requiring, 
as it seemed to you, about so much time and with so much fees 
involved. I could have made a tentative acceptance and then 
come in to look into details, as I did today. 
What details can you look at? 

Sample materials, or descriptions by an observer of the process, 
availability of controls, time constraints etc. What I found 
today made it clear you simply wanted the best that could be 
done in a very limited time, and I took it on that basis 
- details later. Of course, it probably won ^ t answer some of 
the crucial evaluation questions, but to do that you should 
have brought someone in at the beginning. Your best plan would 
have been to send me reasonably typical materials and tell me 
how long the treatment runs. That would have let me form my 
own tentative framework. But no evaluaxor gets perfect condi- 
tions. The trouble is that the ^oss is not his, it's the con- 
sumer's. And that means he's usually not very motivated to 
preserve his objectivity. It's more fun to be on friendly 
terms. with the project peopie. By the way, the project I'm on 
for you is hard to describe concisely in goal-free language, 
but that's not true in all cases. I often do CAI evaluations, 
for example, nnd other educational technology cases, where the 
description of the project isn't goal-loaded. 

L^ok, how long after you've looked at materials before you form 
a pretty good iden about the goals of the project? Isn't it a 
bit absurd to firnt over hearing it a little earlier? 



The important;. qu»^st. vtn i ri.-^t whether 1 do infer the go<T I s but 
whethtT 1 nv\7 irit'^r r,omo >lh^'r possible effects before I am 
locked-in t(^ *set' towar-ds the prr)jecLS' own f^onl.-;. For 
example, I've Looked at elezientnrv r^i;hiOt:l m.iteriais and thoui.^ht 
to nyself - voc-abulary, spelling, general knowledge, two- 
dime :i 3 i -. 'r. r> ; p r e :. n I a r. : r> n r r ir/ e n ion s , b o o k - o r i en t n t i o n , t *ea d - 
in-.": 5:ki:-5, Indenender.i r;, turly caoacity, and r,o on. Tt isn't 
importJint whi!:h t>f tyie.'^e i .v 'he? main ^"!;v.>a I. : T the authors 
have made .-^^riy ^Uf^nl f i.c:ml headway on it, it will show up; I'm 
not li'vely to miss it altogether. And the other dimensions are 
not mas;<ed by your :ret you don't have one. F\emember that 
even ii a sm^^Ie r,:ie-effeot doesn't swamp the intended effect, 
zhf^- tota I, I ly of then may make a very i^eal plus for this pro- 
gramme by comparison wi th others wh i ch do about ^. we "1 1 on the 
i n te-ni-''^d e ff ec t and on cost . A:'ter I ' ve 1 ooked a t ma teria 1 s 
(n<;t i.nc.udinf; teachers' handbooks, etc,\ I look at their 
te5:t:;. r^f course, looking at materials is a little corrupting, 
to.'s \ 1' y;.j want to talk about pure approaches. What I should 
really oe loor'Anr, at is s tuden ts - especially changes in stu- 
dents, and even morr? ei-pecially, changes due to these materials, 
[l*ni cuite happy to be Lov>kin;;; at their' test results, for 
example. ^ Hvit the evaluates usually has to work pretty hard^ 
be f 're he can estab i i sh cause . It' s worth rea i i r.ing , however , 
that if h.e h?d all that, his job is not yet half done. But 1 
f^,urss the m:'-st i rr.p'. - r tan t practical argument for goal -free 
evaluati'-^n is ^n^ we haren': t -uched yet. 
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